vklochan / python-logstash Goto Github PK

View Code? Open in Web Editor NEW

461.0 461.0 137.0 47 KB

Python logging handler for Logstash.

License: MIT License

Python 100.00%

python-logstash's People

Contributors

Stargazers

Watchers

Forkers

moniker-dns alex---- stavxyz kiall nsmfoo shaunduncan javawizard sohonetlabs zanderle daleobrien bigfootproject lvanderree belda aexeagmbh nukemberg jwieland mlvn23 graingert aledotdev rremer yapengwu qudos-com buendiya seekingalpha theho shreyaskarnik egalev enjoyonlineltd suquant parth115 jeffreyvdb grahamhayes dunglan edudar quamilek dnano romantomjak jiajie999 haard kevincolyar coolshop-com skwashd tsvenkat adrtdr matt-snider aadebuger anweshadas intezer ivanitskiy mikodes brenoarosa josemgoujat rychenga sleitner vuchau sallyruthstruik pypestream djsan15 lukehuang slate-studio mrname craiga uishon andreashegergenomics bluetm jxub marrrcin maruf0011 snagajob gowtygs-personal alba-synchrotron bryand1 yuseferi housecanary be-lb toonow weisong82 ducky2048 scp10011 offensity tiggi grzesuav yeseni-today haniffm aweffr eiunkarnewyorker acidjunk alself toby82 nurettin darelvc mirzakhany jfrfonseca pranavgupta1234 guya-ltd jjililic tkern gloomyday macolszewski marcelfenerich

python-logstash's Issues

AMQP handler should support dynamic routing keys

This would be very helpful with topic exchanges

Nested json serialization problem

Got an error with logging inner json:

test_logger.info('python-logstash: test extra fields', extra=extra)

class Test:

    def __init__(self):
        self.a = 1
        self.b = 1

    def __repr__(self):
        return "Test(a={}, b={})".format(self.a, self.b)

test_logger.info("test inner json", extra={"meta": {"test": Test()}})

Gives:

Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 577, in emit
    s = self.makePickle(record)
  File "/home/stas/workspace/partners/python-logstash/logstash/handler_udp.py", line 17, in makePickle
    return self.formatter.format(record)
  File "/home/stas/workspace/partners/python-logstash/logstash/formatter.py", line 142, in format
    return self.serialize(message)
  File "/home/stas/workspace/partners/python-logstash/logstash/formatter.py", line 83, in serialize
    return json.dumps(message)
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Test(a=1, b=1) is not JSON serializable

Tip: Global exception handling

Just had to work this out for myself. If you're dependent on logstash for keeping records, then unhandled exceptions should really be shipped there too. I did this by calling the method below in my setup:

def config_global_exception_handling():
    '''
    Override the global exception hook to come here.
    Here, log the exception which includes sending to logstash.
    '''
    def customexcepthook(exc_type, exc_value, exc_traceback):
        sys.exc_info = lambda: (exc_type, exc_value, exc_traceback)
        logging.exception('Unhandled exception')
    sys.excepthook = customexcepthook

"Unhandled exception" becomes the message attribute, and the stack_trace attribute gets populated appropriately

Support verbose logging

python-logstash/logstash/formatter.py

Line 26 in a8be251

skip_list = (

Because these are stripped down in the base class, it's somewhat problematic to re-extract them for use in debug and production. Things like lineno are particularly useful in logging. I would love to see an easier means to bypass this removal, and include them for logstash.

Would be very handy if this was just a flag passed down through the LogstashHandlers.

PyPI release for 0.4.5

Hi, the current version on pypi is 0.4.2. Following the installation instructions (pip install python-logstash) will install that version.

It would be very nice if you could make a new release to pypi, so that it is possible to install the latest version from it.

Thanks!

Log entries with nested dicts get silently ignored

I am using this logger for a Django project and it seems that whenever I try to log a nested JSON object, it gets silently ignored. The following works fine:

logger.debug('trace', extra={
    # 'request': {
    #     'path': request.get_full_path()
    # },
    'path': request.get_full_path(),
})

But this one gets silently ignored:

logger.debug('trace', extra={
    'request': {
        'path': request.get_full_path()
    },
    'path': request.get_full_path(),
})

Any ideas?

upload 0.4.7 on pypi

on pypi last version of package is 0.4.6

0.4.6 version have no ssl feature as I can see... :(

No Error handling or message for wrong Port

My Logstasah is listening TCP on 5000, and when trying you sample

import logging
import logstash
import sys

host = 'localhost'

test_logger = logging.getLogger('python-logstash-logger')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash.LogstashHandler(host, 5959, version=1))
# test_logger.addHandler(logstash.TCPLogstashHandler(host, 5959, version=1))

test_logger.error('python-logstash: test logstash error message.')
test_logger.info('python-logstash: test logstash info message.')
test_logger.warning('python-logstash: test logstash warning message.')

# add extra field to logstash message
extra = {
    'test_string': 'python version: ' + repr(sys.version_info),
    'test_boolean': True,
    'test_dict': {'a': 1, 'b': 'c'},
    'test_float': 1.23,
    'test_integer': 123,
    'test_list': [1, 2, '3'],
}
test_logger.info('python-logstash: test extra fields', extra=extra)

Nothing Happened , no error , nothing
Even if I changed the port to 500 still nothing.

but simple nc localhost 5000 < /var/log/test.log
it works.

Clarification Required: BaseLogstash Handler i.e (LogstashHandler) is equivalent to the UDPLogstashHander?

This more me being a noob, but I am guessing that BaseLogstash Handler is the same as the UDP Handler defined? (I'm new to python :P)

https://github.com/vklochan/python-logstash/blob/master/logstash/handler_udp.py#L21

Sorry for any inconvenience :P

Setting formatter tags

Formatter class has tags argument, but handler class always sets tags to empty list.
Am i missing something or there is no way to set tags if you configure logging with dictConfig (like in Django)?

unicode support in formatter.py

Hello,

Name: python-logstash
Version: 0.4.6
Summary: Python logging handler for Logstash.
Home-page: https://github.com/vklochan/python-logstash
Author: Volodymyr Klochan
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib/python2.7/dist-packages
Requires:

Using python-logstash through ansible logstash call back I get the following error:

TASK [java/jdk : Préparation répertoire destination Java] ***********************************************************************************************************************************************************************************
changed: [tnr-container]
TASK: java/jdk : Préparation répertoire destination Java
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 578, in emit
    s = self.makePickle(record)
  File "/usr/local/lib/python2.7/dist-packages/logstash/handler_tcp.py", line 25, in makePickle
    return self.formatter.format(record) + b'\n'
  File "/usr/local/lib/python2.7/dist-packages/logstash/formatter.py", line 136, in format
    message.update(self.get_extra_fields(record))
  File "/usr/local/lib/python2.7/dist-packages/logstash/formatter.py", line 45, in get_extra_fields
    fields[key] = repr(value)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 19: ordinal not in range(128)
Logged from file logstash.py, line 131

The problem comes from the latin1 character present in the handled value.

logstash server restart handling

What should be done from a python client when a Logstash server is restarted?

I'm running a Logstash v2.3.4 container on my local post via a docker image found on docker hub. https://hub.docker.com/_/logstash/

logstash.conf

input {
  tcp {
    port => 5000
    codec => json
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

I'm also running a Flask app on my localhost with python-logstash connecting via TCP.

When I restart the container that is running Logstash I no longer see logged messages being displayed to stdout until I restart my Flask server. If anyone can help illuminate what should be done here, or else point me in the right direction, then that would be appreciated.

Broken example on list with elements of different types

hi
in the example in the README

extra = {
    'test_string': 'python version: ' + repr(sys.version_info),
    'test_boolean': True,
    'test_dict': {'a': 1, 'b': 'c'},
    'test_float': 1.23,
    'test_integer': 123,
    'test_list': [1, 2, '3'],
}

you mention a list 'test_list': [1, 2, '3'],, where elements have different types ==> THIS IS NOT WORKING!

Import Event error

../../lib/logger/elkhandler.py:5: in
import logstash
lib/python3.4/site-packages/logstash/init.py:2: in
from event import Event
E ImportError: No module named 'event'

I have installed events and python-logstash using pip and both are satisfied.. I'm using python
python --version
Python 3.4.3

pip install events
Requirement already satisfied: events in /lib/python3.4/site-packages
pip install python-logstash
Requirement already satisfied: python-logstash in /lib/python3.4/site-packages

if python-logstash support domain unix sock

if python-logstash support send to domain unix sock?

Inconsistency in LogstashFormatterBase.serialize method for python 3

For python 3 LogstashFormatterBase.serialize method returns bytes instead of string. This incosistency results in no possibility in reusing this formatter with buildin handlers like pythons StreamHandlers (e.g. FileHandler), because it expects string object returned from formatter.

For python 2.X LogstashFormatterBase.serialize method behaves correct returning string.

Logs are not in order if `@timestamp` is the same

Hi, thanks for this library, so I have some consecutive logs that have the same @timestamp but looking in Kibana, they are not in the order that they are printed in the code

For example, the first and last logs are in the right order here, but the two in the middle are not:

Has this happened to anyone else and is there a possible fix for this?

Please document that the project is not using semantic versioning.

Given that this is a utility library, it would make sense to follow semantic versioning.

In fact, the change in 0.4.6 (renaming field names) changed the external API. We mistakenly believed that python-logstash would follow semantic versioning and pinned our dependencies using python-logstash >= 0.4.2, <1.0 so the 0.4.6 update was included in our updates.

This has now created a bunch of entries in our ELK stack with the wrong field names and our visualisations now show incorrect information. And this was only noticed by pure chance.

It would be nice if the versioning scheme used in python-logstash would be documented so nobody else falsely assumes that this is semver.

Message nested inside message

Message i log is nested inside dictionary with key "message", together with other python-logstash values. Here's raw content i get in kibana:

{
  "_index": "logstash-2014.07.06",
  "_type": "syslog",
  "_id": "Vshw1docTteonciCQktJZA",
  "_score": null,
  "_source": {
    "message": "{\"host\": \"HOSTNAME\", \"logger\": \"test.log\", \"type\": \"logstash\", \"tags\": [], \"path\": \"C:\\\\Users\\\\user\\\\file.py\", \"@timestamp\": \"2014-07-06T07:51:26.004000Z\", \"foo\": \"bar\", \"@version\": 1, \"message\": \"Hello world\", \"levelname\": \"INFO\"}",
    "@version": "1",
    "@timestamp": "2014-07-06T07:46:31.445Z",
    "type": "syslog",
    "tags": [
      "syslog",
      "_grokparsefailure"
    ],
    "host": "10.100.1.1",
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice"
  },
  "sort": [
    1404632791445,
    1404632791445
  ]
}

I configured logger more or less like example in readme file.
What am i doing wrong?

Examples in the readme do not match up

I was running through how to set this up, most specifically the receiving part in logstash.

I noticed in your examples you set the handler's target host port to 5959, but in the example logstash configuration it is port 5000.

Additionally, it's mentioned we're sending messages to logstash using UDP, but the logstash configuration example is for a TCP input.

Error Handling of TCPLogstashHandler When Writing to Logstash

I added "try... except..." to example1.py of this repo as following:

import logging
import logstash
import sys

host = 'localhost'

try:
	test_logger = logging.getLogger('python-logstash-logger')
	test_logger.setLevel(logging.INFO)
	# test_logger.addHandler(logstash.LogstashHandler(host, 5959, version=1))
	test_logger.addHandler(logstash.TCPLogstashHandler(host, 5959, version=1))

	test_logger.error('python-logstash: test logstash error message.')
	test_logger.info('python-logstash: test logstash info message.')
	test_logger.warning('python-logstash: test logstash warning message.')

	# add extra field to logstash message
	extra = {
	    'test_string': 'python version: ' + repr(sys.version_info),
	    'test_boolean': True,
	    'test_dict': {'a': 1, 'b': 'c'},
	    'test_float': 1.23,
	    'test_integer': 123,
	    'test_list': [1, 2, '3'],
	}
	test_logger.info('python-logstash: test extra fields', extra=extra)

	print('Succeed to save event.')


except Exception as e:
	print('Failed to save event: ',e)

Even though no Logstash on my localhost, it will say "Succeed to save event", I am wondering how to add error handling of writing to Logstash with TCPLogstashHandler. Thanks.

Feature Request: Add level number/value to the logged fields.

Currently, the emitted messages only contain level names (f.ex. ERROR, CRITICAL, ...). Those names correspond to level values. Those names can be converted to/from their numeric values using logging.getLevelName. Additionally, the numeric values should be available on the log-record behind the attribute levelno:

>>> from logging import LogRecord
>>> import logging
>>> r = LogRecord('foo', logging.DEBUG, 'path', 11, 'msg', {}, False)
>>> r.levelno
10
>>> r.levelname
'DEBUG'

Adding the numeric level value makes it much easier to filter on the severity by simply filtering on everything greater than 40 for example.

@version field should be string not int

Hi,
In Elasticsearch the '@Version' field is a string, but because python-logstash sends it as an integer, it is being saved as integer in ES.

Suggested is to send the field's value as string.

Thanks,
Yarden

AMQPHandler creates a recursive loop of log messages

Since pika uses python logging framework internally, if the root logger is configured with AMQPHandler then a loop is created where pika logs are sent through the amqp handler thus causing more logs to be emitted.

Add support for async pika adapters

We would like to use the amqp logging handler within tornado, it would be nice if python-logstash supported pika connection adapters.

Should stack_info be added to skip list?

Here are two other similar libraries, and they both maintain a similar list of blacklisted keys which would conflict with the python log record stuff:

https://github.com/marselester/json-log-formatter/blob/master/json_log_formatter/__init__.py

https://github.com/madzak/python-json-logger/blob/master/src/pythonjsonlogger/jsonlogger.py

The one notable field that those have which this library doesn't have is stack_info, so I wanted to ask whether this is something that should be added.

It seems this was added to the log record in some python version

https://github.com/python/cpython/blob/b7d62050e7d5fc208ae7673613da4f1f2bc565c4/Lib/logging/__init__.py#L336

Its appearance there seems to be comparable to the other fields

BUG: Logstash in Python 3, byte to string in handler_tcp.py

The following error is thrown when configuring logstash for Gunicorn.

Traceback (most recent call last):                                                                                          
  File "/usr/lib/python3.5/logging/handlers.py", line 621, in emit                                                          
    s = self.makePickle(record)                                                                                             
  File "~/Projects/env/lib/python3.5/site-packages/logstash/handler_tcp.py", line 25, in makePickle    
    return self.formatter.format(record) + b'\n'                                                                            
TypeError: Can't convert 'bytes' object to str implicitly

Changing handler_tcp.py to:

from logging.handlers import DatagramHandler, SocketHandler
from logstash import formatter


# Derive from object to force a new-style class and thus allow super() to work
# on Python 2.6
class TCPLogstashHandler(SocketHandler, object):
    """Python logging handler for Logstash. Sends events over TCP.
    :param host: The host of the logstash server.
    :param port: The port of the logstash server (default 5959).
    :param message_type: The type of the message (default logstash).
    :param fqdn; Indicates whether to show fully qualified domain name or not (default False).
    :param version: version of logstash event schema (default is 0).
    :param tags: list of tags for a logger (default is None).
    """

    def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
        super(TCPLogstashHandler, self).__init__(host, port)
        if version == 1:
            self.formatter = formatter.LogstashFormatterVersion1(message_type, tags, fqdn)
        else:
            self.formatter = formatter.LogstashFormatterVersion0(message_type, tags, fqdn)

    def makePickle(self, record):
        return str.encode(self.formatter.format(record)) + b'\n'

Specifically the last line:

return str.encode(self.formatter.format(record)) + b'\n'

Encoding the string to bytes allows the library to run.

Didn't want to submit a pull request in case you or someone else had a better solution.

dropped extra parameter from sent json

tl;dr the extra parameter in logger.info(mesg, extra) is dropped from the sent json request. Why?

problem

Given the following code

#!/usr/bin/env python3.4

import os
import logging
import logstash

# logging init
logger = logging.getLogger(os.path.basename(__file__))
logger.setLevel(logging.INFO)
# logging logstash
logger.addHandler(logstash.TCPLogstashHandler('localhost', 5000, version=1))

mesg = 'hello'
extra = {'test1': 1}
logger.info(mesg, extra)

Capturing the data sent over the wire using tcpdump or wireshark proves 'test1': 1 is not present.

{
  "level": "INFO",
  "stack_info": null,
  "message": "hello",
  "type": "logstash",
  "path": "/tmp/logstash-examples/exampleIssue.py",
  "tags": [],
  "host": "myhost",
  "logger_name": "exampleIssue.py",
  "@timestamp": "2017-02-18T03:26:13.432Z",
  "@version": "1"
}

Stepping through the code shows the extra parameter is skipped for attaching to the record (a dict) within LogstashFormatterBase.get_extra_fields. extra becomes referenced by object attribute args within (.../Lib/logging/__init__.py)

def __init__(self, name, level, pathname, lineno,
             msg, args, exc_info, func=None, sinfo=None, **kwargs):

    # ...

    self.args = args

And then args skipped for adding within LogstashFormatterBase.get_extra_fields because 'args' is in skip_list

class LogstashFormatterBase(logging.Formatter):

    # ...

    def get_extra_fields(self, record):
        # The list contains all the attributes listed in
        # http://docs.python.org/library/logging.html#logrecord-attributes
        skip_list = (
            'args', 'asctime', 'created', 'exc_info', 'exc_text', 'filename',
            'funcName', 'id', 'levelname', 'levelno', 'lineno', 'module',
            'msecs', 'msecs', 'message', 'msg', 'name', 'pathname', 'process',
            'processName', 'relativeCreated', 'thread', 'threadName', 'extra',
            'auth_token', 'password')

        # ...

        fields = {}

        for key, value in record.__dict__.items():
            if key not in skip_list:
                if isinstance(value, easy_types):
                    fields[key] = value
                else:
                    fields[key] = repr(value)

From the call stack

get_extra_fields (.../Lib/site-packages/python_logstash-0.4.6-py3.4.egg!/logstash/formatter.py)
format (.../Lib/site-packages/python_logstash-0.4.6-py3.4.egg!/logstash/formatter.py)
makePickle (.../Lib/site-packages/python_logstash-0.4.6-py3.4.egg!/logstash/handler_tcp.py)
emit (.../Lib/logging/handlers.py)
handle (.../Lib/logging/__init__.py)
callHandlers (.../Lib/logging/__init__.py)
handle  (.../Lib/logging/__init__.py)
_log  (.../Lib/logging/__init__.py)
info  (.../Lib/logging/__init__.py)

solution

As a module user, the current behavior is unexpected, confusing, and prevents me from passing along required extra information. I reviewed logging.html#logrecord-attributes. It was not clear why args attribute (extra parameter) should be effectively dropped. I suggest changing the current behavior to not drop extra parameter information.
If the current behavior must be kept and this problem is to be worked around then moving skip_list to be class instance variable would allow the user to remove 'args' from skip_list as needed.

Using python-logstash 0.4.7 (a8be251ad9fc102af668c38a5397515b0804da75) on cython 3.4.1.

How to limit the number of UDP sockets opened ?

Because, I am running into I/O Error : too many files opened.
And when I do, lsof| grep pid, I see a lot of UDP connections opened and eventually I exhaust on the ulimit number. As soon as the ulimit number is exhausted, and more UDP sockets need to be opened for logging to log stash, I/O Error : too many files opened is encountered.

Raising ulimit doesn't completely fix this. Please help.

Release 0.4.6 to PyPI

I'm anxiously waiting for a new version of python-logstash to be released with #41 included. I see that the version in setup.py has already been bumped. Are you going to push it to PyPI soon?

distutils has been deprecated in Python 3.10

https://www.python.org/dev/peps/pep-0632/#migration-advice

Using setuptools.setup might help here.

python-logstash/setup.py

Line 1 in 7bd98c0

from distutils.core import setup

Set name

Hi there,

How can I set the name or an extra field fixed for all log messages? I'm trying to use a generic input/output having a variable for index, so I need to set the name or have a field on my python app.

output {
	elasticsearch {
	    index => "%{name}"
		hosts => "elasticsearch:9200"
	}
}

SSL Support?

I want to know if python-logstash has SSL support, because i cannot find any way to define custom certificate.

Thanks

Error installing python-logstash via pip

CentOS 7

$ pip version
pip 9.0.1 from /usr/lib/python2.7/site-packages (python 2.7)

$ pip install python-logstash
Collecting python-logstash
  Using cached python-logstash-0.4.6.tar.gz
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 412, in run_egg_info
    self.setup_py, self.name,
  File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 12, in <module>
    import setuptools.version
  File "/usr/lib/python2.7/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 72, in <module>
    import packaging.requirements
  File "/usr/lib/python2.7/site-packages/packaging/requirements.py", line 59, in <module>
    MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker")
TypeError: __call__() takes exactly 2 arguments (1 given)

I'm not exactly sure what to do with a pip install error. I'm a gem/cpan guy.

Example Logstash config

An example Logstash config would be helpful to get started easier.

I will post my own as soon as I have a working one - but maybe the author also could post his/add it to the README? :-)

AttributeError: module 'logstash' has no attribute 'LogstashHandler'

my Logstash is up and running in my system but when I wanted to test you sample code faced with

AttributeError: module 'logstash' has no attribute 'LogstashHandler'

while I've installed logstash package in my system.

python version : 3.6

How can we add authentication to logstash?

Formatting of extra fields, repr() vs str()

python-logstash/logstash/formatter.py

Line 45 in a8be251

fields[key] = repr(value)

The value of extra fields are formatted via repr(). In most cases it seems to me that str() would be better. A typical example is Path objects:

from pathlib import Path
path = Path("foo/bar")
print(repr(path))
print(str(path))

The repr-version is PosixPath('foo/bar'), whereas the str-version is foo/bar. In logs, I always want to have the latter version and, currently, have to manually enforce this via extra={"path": str(path)}. This is also true for many other objects.

According to Python docs repr() should return "the “official” string representation of an object" whereas str() should return a "nicely printable string representation of an object". For logs, I think, the latter is almost always what one wants.

Therefore my question: would you be open to changing the formatting of extra fields from repr() to str()? Am I missing some reason why it is a bad idea? If you agree, I would submit a PR.

Add a TCP handler

I imagine this would be as simple as creating a LogstashHandler equivalent that subclasses directly from logging.SocketHandler.

why ? python-logstash ' json have not grok effect ...

I thank message'field is field,not json string

from kibana

@timestamp 2015-03-30T05:09:42.151Z
@Version 1
_id AUxpFdtj8ZYGI1FQ4LPB
_index logstash-2015.03
_type logs
host 192.168.1.50
message {"nima": "ruifengyun", "message": {"@source": "logstash://bj-log-1/run.py", "source": "ruifengyun", "@source_host": "bj-log-1", "@message": "python-logstash: test extra fields", "@tags": [], "@fields": {"test_integer": 123, "nima": "ruifengyun", "test_dict": {"a": 1, "b": "c"}, "test_boolean": true, "test_list": [1, 2, "3"], "test_float": 1.23, "logger": "python-logstash-logger", "levelname": "INFO", "test_string": "ruifengyun(2, 6, 6, 'final', 0)"}, "@timestamp": "2015-03-30T05:09:42.150821Z", "@source_path": "run.py", "@type": "logstash"}}

Allow "tags" to be specified in "extra" for log messages.

I have an application which needs to send messages with different tags to logstash. The tags are used to classify messages and change over runtime. It seems wasteful to me to generate a new handler instance just to support tags.

For one execution I expect to be in the tens for tags (so not that many). But I can't say from the beginning which tags will be emitted so sending a message looks like this currently:

def send_message(tags, message):
    handler = TCPLogstashHandler([...], tags=tags)
    LOG.addHandler(handler)
    LOG.info(message)
    LOG.removeHandler(handler)

Having the tags exposed in the via the "extra" attribute would allow me to write this:

def send_message(tags, message):
    LOG.info(message, extra={'tags': tags}

which would even make the send_message function redundant.

Throwing TypeError when logging

I have a StackOverflow question, but it hasn't gotten any attention, so I thought I'd bring it up here.

The requirement:

Create a logger that can log the hostname as a part of the formatter. The logger will have many handlers, one of which is a logstash handler.

The code:

import logging
import logstash

from socket import gethostname

class CustomLogger(logging.Logger):
    def _log(self, level, msg, args, exc_info=None, extra=None):
        if extra is None:
            extra = { 'hostname' : gethostname() }
        super(CustomLogger, self)._log(level, msg, args, exc_info, extra)

def setup_custom_logger(host, port):
    # add hostname to the formatter.
    logging.setLoggerClass(CustomLogger)

    formatter = logging.Formatter(fmt='%(hostname)s - %(asctime)s - %(levelname)s - %(module)s - %(message)s')

    logstash_handler = logstash.LogstashHandler(host, port, version=2)
    logstash_handler.setLevel(logging.DEBUG)
    logstash_handler.setFormatter(formatter)

    logger = logging.getLogger(__name__)
    logger.setLevel(logging.DEBUG)
    logger.addHandler(logstash_handler)

    return logger

The issue:

Logging anything results in:

--- Logging error ---
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/logging/handlers.py", line 620, in emit
    self.send(s)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/logging/handlers.py", line 678, in send
    self.sock.sendto(s, self.address)
TypeError: a bytes-like object is required, not 'str'

I've tried logger.info('hello'), logger.info(b'hello'), logger.info(1) -- they always result in a TypeError.

Does anyone have an idea why this isn't working? I'm using 0.4.6.

UDPHandler is missing newline

makePickle function is missing (+ b'\n') at the end.

Adopt standard field names

Check out https://github.com/logstash/logstash-logback-encoder#standard-fields or https://github.com/logstash/log4j-jsonevent-layout#usage. Both are supported by logstash crew.

Mismatches are minor but having several names for the same field is no fun as well.
logger_name vs logger
level vs levelname

Feature Request > option to add extra fields to records globally

I find my self doing this:

class CustomUDPLogstashHandler(logstash.UDPLogstashHandler):
    def __init__(self, *a, **kw):
        self.global_extras = kw.pop('global_extras', {})
        super().__init__(*a, **kw)

    def emit(self, record):
        for key, value in self.global_extras.items():
            setattr(record, key, value)
        super().emit(record)

LOGGING={
...
        'logstash': {
            'level': 'DEBUG',
            'class': 'util.CustomUDPLogstashHandler',
            'host': 'logstash.service',
            'port': 5000,
            'version': 1,
            'tags': ['tag1', 'tag2'],
            'global_extras': {
                'product': "my_test_product",
                'instance': "my_test_instance",
                'site': 333
            }
        }
...

It would be cool if that functionality were included

log formatting error with string and bytes concatenation

About

When I want to format logs I face error[Appendix A] telling that concatenate of str and bytes is impossible, but it works when I remove parameter: formatter=text in my log.ini file[Appendix B]

Appendix A | traceback

"Waiting for application startup."
--- Logging error ---
Traceback (most recent call last):
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/logging/handlers.py", line 673, in emit
    s = self.makePickle(record)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/logstash/handler_tcp.py", line 25, in makePickle
    return self.formatter.format(record) + b'\n'
TypeError: can only concatenate str (not "bytes") to str
Call stack:
  File "<string>", line 1, in <module>
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
    return self._bootstrap(parent_sentinel)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/base_events.py", line 633, in run_until_complete
    self.run_forever()
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/base_events.py", line 600, in run_forever
    self._run_once()
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/base_events.py", line 1896, in _run_once
    handle._run()
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/server.py", line 77, in serve
    await self.startup(sockets=sockets)
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/server.py", line 88, in startup
    await self.lifespan.startup()
  File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 45, in startup
    self.logger.info("Waiting for application startup.")

Appendix B | `log.ini` config file

[handler_logstash]
class=logstash.TCPLogstashHandler
level=INFO
formatter=text
args=('127.0.0.1', '8080', '1')

[formatter_text]
format="%(message)s"
datefmt='%d-%m-%Y %H:%M:%S'

Change json format to work with Logstash 1.2.x

python-logstash has to be modified to work with the new JSON format in logstash 1.2.x. The change simply involves removing the "@" signs from all the dictionary keys.

ResourceWarning: unclosed <socket ...>

When using the TCPLogstashHandler and sending a message, I get the following warning:

ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.27.0.74', 36854), raddr=('172.27.0.63', 5001)>

I'm not certain if this comes from the underlying SocketHandler from Python or from python-logstash. If it's the former, feel free to close this issue.

1000ms latency caused by unused config

Hey there, we had some old config sitting around pointing to a localhost logstash – which did not exist.

It turns out that this was causing 1000ms latency when calling logger.info for the first time.

I don't understand the root cause, but I just thought I would note here that this seems like a silent failure mode which could be fixed.

Related django config

            "logstash": {
                "level": "INFO",
                "class": "logstash.TCPLogstashHandler",
                "host": os.getenv("LOGS_IP", "localhost"),
                "port": 5000,  # Default value: 5000
                "version": 1,
                "message_type": "django_logstash",  # 'type' field in logstash message.
                "fqdn": False,  # Fully qualified domain name. Default value: false.
                # 'tags': ['django.request'],  # list of tags. Default: None.
            },

Gunicorn log handler for logstash not working

As per gunicorn documentation it is accepting the standard Python logging module’s Configuration file format.

but how can I give logstash handler to gunicorn in configuration file , what is the syntax for that?

I gave like below:

[loggers]
keys=root, logstash

[handlers]
keys=console , logstash

[formatters]
keys=generic, access

[logger_root]
level=INFO
handlers=console

[logger_logstash]
level=DEBUG
handlers=logstash
propagate=1
qualname=logstash


[handler_console]
class=StreamHandler
formatter=generic
args=(sys.stdout, )

[handler_logstash]
class=logstash.TCPLogstashHandler
formatter=generic
args=('localhost',5959)


[formatter_generic]
format=%(asctime)s [%(process)d] [%(levelname)s] %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter

[formatter_access]
format=%(message)s
class=logging.Formatter

not working