greyzmeem / python-logging-loki

Python logging handler for Loki

python-logging-loki

Python logging handler for Loki.
https://grafana.com/loki

Installation

pip install python-logging-loki

Usage

import logging
import logging_loki


handler = logging_loki.LokiHandler(
    url="https://my-loki-instance/loki/api/v1/push", 
    tags={"application": "my-app"},
    auth=("username", "password"),
    version="1",
)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)
logger.error(
    "Something happened", 
    extra={"tags": {"service": "my-service"}},
)

Example above will send Something happened message along with these labels:

• Default labels from handler
• Message level as serverity
• Logger's name as logger
• Labels from tags item of extra dict

The given example is blocking (i.e. each call will wait for the message to be sent).
But you can use the built-in QueueHandler and QueueListener to send messages in a separate thread.

import logging.handlers
import logging_loki
from multiprocessing import Queue


queue = Queue(-1)
handler = logging.handlers.QueueHandler(queue)
handler_loki = logging_loki.LokiHandler(
    url="https://my-loki-instance/loki/api/v1/push", 
    tags={"application": "my-app"},
    auth=("username", "password"),
    version="1",
)
logging.handlers.QueueListener(queue, handler_loki)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)
logger.error(...)

Or you can use LokiQueueHandler shortcut, which will automatically create listener and handler.

import logging.handlers
import logging_loki
from multiprocessing import Queue


handler = logging_loki.LokiQueueHandler(
    Queue(-1),
    url="https://my-loki-instance/loki/api/v1/push", 
    tags={"application": "my-app"},
    auth=("username", "password"),
    version="1",
)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)
logger.error(...)

python-logging-loki's Issues

Emitter raises TypeError when trying to build tags

When trying to make tags deepcopy, the TypeError: can't pickle _thread.RLock objects error may occur, since self.tags is an instance of ConvertingDict.

python-logging-loki/logging_loki/emitter.py

Line 90 in a82bb6d

tags = copy.deepcopy(self.tags)

SSL Error: Wrong Version Number

Hello. I'm getting the following error when trying run the default example found in the documentation for this package.

Traceback (most recent call last):
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/logging_loki/handlers.py", line 81, in emit
devops-tools             |     self.emitter(record, self.format(record))
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/logging_loki/emitter.py", line 55, in __call__
devops-tools             |     resp = self.session.post(self.url, json=payload)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/requests/sessions.py", line 635, in post
devops-tools             |     return self.request("POST", url, data=data, json=json, **kwargs)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
devops-tools             |     resp = self.send(prep, **send_kwargs)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
devops-tools             |     r = adapter.send(request, **kwargs)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/requests/adapters.py", line 563, in send
devops-tools             |     raise SSLError(e, request=request)
devops-tools             | requests.exceptions.SSLError: HTTPSConnectionPool(host='devops-tools-loki', port=3100): Max retries exceeded with url: /loki/api/v1/push (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)')))
devops-tools             | Call stack:
devops-tools             |   File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
devops-tools             |     self._bootstrap_inner()
devops-tools             |   File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
devops-tools             |     self.run()
devops-tools             |   File "/usr/local/lib/python3.8/threading.py", line 870, in run
devops-tools             |     self._target(*self._args, **self._kwargs)
devops-tools             |   File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
devops-tools             |     work_item.run()
devops-tools             |   File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
devops-tools             |     result = self.fn(*self.args, **self.kwargs)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
devops-tools             |     return func(*args, **kwargs)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
devops-tools             |     response = response or self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
devops-tools             |     response = get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/ms_identity_web/django/middleware.py", line 32, in __call__
devops-tools             |     response = self.get_response(request)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/asgiref/sync.py", line 202, in __call__
devops-tools             |     current_executor.run_until_future(loop_future)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/asgiref/current_thread_executor.py", line 62, in run_until_future
devops-tools             |     work_item.run()
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/asgiref/current_thread_executor.py", line 22, in run
devops-tools             |     result = self.fn(*self.args, **self.kwargs)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
devops-tools             |     return func(*args, **kwargs)
devops-tools             |   File "/opt/meraki_gpns_tools/automationproject/decorators.py", line 65, in wrap
devops-tools             |     return function(request, user=user_object, *args, **kwargs)
devops-tools             |   File "/opt/meraki_gpns_tools/automationproject/decorators.py", line 28, in wrap
devops-tools             |     return function(request, api_key=meraki_key, *args, **kwargs)
devops-tools             |   File "/opt/meraki_gpns_tools/unonetapp/views.py", line 86, in onboardingView
devops-tools             |     logger.error(
devops-tools             |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1475, in error
devops-tools             |     self._log(ERROR, msg, args, **kwargs)
devops-tools             |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1589, in _log
devops-tools             |     self.handle(record)
devops-tools             |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1599, in handle
devops-tools             |     self.callHandlers(record)
devops-tools             |   File "/usr/local/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
devops-tools             |     hdlr.handle(record)
devops-tools             |   File "/usr/local/lib/python3.8/logging/__init__.py", line 954, in handle
devops-tools             |     self.emit(record)
devops-tools             |   File "/root/.local/share/virtualenvs/meraki_gpns_tools-7f3ZpwMf/lib/python3.8/site-packages/logging_loki/handlers.py", line 83, in emit
devops-tools             |     self.handleError(record)
devops-tools             | Message: 'Something happened'
devops-tools             | Arguments: ()

Apparently I'm using a wrong SSL version, but I dont have the slightest idea of what that means.

Multi-tenant support

We need to sent logging to a multi-tenant Loki system from custom build python apps, but multi-tenancy is not supported in the code.

I have already created a fix which is working, but cannot push the branch. Could you provide me with the right WoW for submitting these kind of changes?

Loss of timestamp precision when formatting to rfc3339

In the emit function you convert the created attribute of the logrecord to rfc3339,

ts = rfc3339.format(record.created)

Having looked at the source of the rfc3339 package this actually loses you some precision. It might be wise to consider an alternative to it.

Issues when using with Celery

I have the following method that runs as a background task:

import logging

from config.celery import app
from config.queues import my_queue

logger = logging.getLogger(__name__)


@app.task(name='my-method', queue=my_queue)
def my_method(**kwargs) -> None:
    # Method body ...
    logger.info('This is my log message!', extra={
        'tags': {
            'foo': 'bar',
        },
    })

Calling this method anywhere as:

my_method.delay(**kwargs)

will result Celery disconnecting from the broker (which is RabbitMQ) with the following WARNING message:

rabbitmq       | 2022-05-23 06:51:06.177726+00:00 [warning] <0.1438.0> closing AMQP connection <0.1438.0> (172.21.0.1:38034 -> 172.21.0.4:5672, vhost: '/', user: 'guest'):
rabbitmq       | 2022-05-23 06:51:06.177726+00:00 [warning] <0.1438.0> client unexpectedly closed TCP connection

Furthermore, the message is also not sent to Loki.
Here is my Loki settings:

LOKI_SETTINGS = {
    'level': 'INFO',
    'class': 'logging_loki.LokiHandler',
    'url': LOKI_URL,
    'tags': {
        'app': 'my-app'
    },
    'auth': (LOKI_USER, LOKI_PASSWORD),
    'version': '1',
}

And here is my Django logging setting:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
        },
        'loki': LOKI_SETTINGS,
    },
    'loggers': {
        '': {
            'handlers': ['console', 'loki'],
            'level': 'DEBUG',
            'propagate': True,
        }
    }
}

I'm using Django==3.2, celery==5.1.2, and python-logging-loki==0.3.1.
My RabbitMQ is also version 3.9.16.

Thanks!

Time ns is NaN in grafana

the function build_payload in class LokiEmitterV1 create a ts. and the session.post return 204.
however, time ns in grafana is Nan

V1 timestamp

Should the V1 emitter timestamp be retrieved from record.created, just like it used to be in V0?

This is how it was set for a V0 record and there seems to be no reason why this shouldn't also apply to V1. The divergence makes a migration more difficult due to the inconsistent handling.

Suggestion for change:

ts = rfc3339.format_microsecond(record.created)

Happy to submit a pull request with the change, if contributors and @GreyZmeem agree.

No Unique Label

I am pushing logs to loki and it is working nice.
But when I show the log in Grafana, it is saying "no unique labels"
Could you guide me how I can set the unique label?

Extra tags cause Loki to internal error

I tried the following

import logging
import logging_loki

handler = logging_loki.LokiHandler(
    url="https://$MYSERVER/api/prom/push",
    tags={"application": "bruno-test"},
)

logger = logging.getLogger("bruno-logger")
logger.addHandler(handler)
logger.error(
    "Something happened",
)

and that works:

but I do:

logger.error(
    "Something happened",
    extra={"tags": {
        "bruno-tag": "xxx"
    }},
)

I get

--- Logging error ---
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/logging_loki/handlers.py", line 72, in emit
    raise ValueError("Unexpected Loki API response status code: %s" % resp.status_code)
ValueError: Unexpected Loki API response status code: 500
Call stack:
  File "a.py", line 14, in <module>
    "bruno-tag": "xxx"
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1407, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1514, in _log
    self.handle(record)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 1586, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.7/logging/__init__.py", line 894, in handle
    self.emit(record)
  File "/usr/local/lib/python3.7/site-packages/logging_loki/handlers.py", line 74, in emit
    self.handleError(record)

on Loki server side:

level=warn ts=2019-09-25T05:34:17.971104105Z caller=logging.go:49 traceID=5ff3ee51249a93 msg="POST /api/prom/push (500) 242.786µs Response: \"parse error at line 1, col 71: syntax error: unexpected IDENTIFIER, expecting = or != or =~ or !~\\n\" ws: false; Accept: */*; Accept-Encoding: gzip, deflate; Content-Length: 205; Content-Type: application/json; User-Agent: python-requests/2.22.0; X-Forwarded-For: 10.164.0.21; X-Forwarded-Host: $MYSERVER; X-Forwarded-Port: 443; X-Forwarded-Proto: https; X-Forwarded-Server: traefik-7856fd8df5-n4t8d; X-Real-Ip: 10.164.0.21; "

i'm running loki official build from docker image grafana/loki:v0.3.0

Should only use log level higher than DEBUG with the handlers

The instructions should state that setLevel() should be called on the handlers (LokiHandler, LokiQueueHandler) with a level higher than DEBUG.

The reason is that the handlers eventually call/use code in urllib3, (to send the log messages to the Loki API) which itself generates log DEBUG messages. If the log handler processes these messages, it results in additional calls to the urllib3 code, which generates further log messages ad infinitum, potentially causing an infinite loop of log messages if the Python process is not terminated.

Ideally the handler should ignore log messages from any code it directly or indirectly calls, but for now you should always use setLevel() with something higher than logging.DEBUG on the handler.

Tracing not showing in grafana

Hi Folks,

I am trying to use Loki + Tempo but when I check my logs in Loki the trace_id and span_id are missing.

Code related to logging:

[...]
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from logging_loki import LokiHandler, emitter

emitter.LokiEmitter.level_tag = "level"

LoggingInstrumentor().instrument(set_logging_format=True)
handler = LokiHandler(
    url="http://ipadpi.local:3100/loki/api/v1/push",
    tags={"application": "backend"},
    version="1",
)
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)

[...]

@app.get("/test")
async def test():
    logger.info(
        "Info em test2",
        extra={"tags": {"service": "backend"}},
    )

    return Status(message='test!')

The result in loki is:

and I am running my uvicorn using:
opentelemetry-instrument uvicorn main:app

What am I missing here?

Can one log to loki with a time in the past?

I have an agent that sometimes detects elements that need to be logged "into the past".

python-logging-loki reuses the python-logging mechanism, so I tried adding a timestamp filter as specifief in the stackoverflow message (see link in source code).

Can one 'tweak' the datetime on which a log line is sent to loki?

import logging
import logging_loki
import time

# https://stackoverflow.com/questions/47146514/python-logging-using-custom-timestamps-not-current-timestamp

class TimestampFilter (logging.Filter):
        """
        This is a logging filter which will check for a `timestamp` attribute on a
        given LogRecord, and if present it will override the LogRecord creation time
        to be that of the timestamp (specified as a time.time()-style value).
        This allows one to override the date/time output for log entries by specifying
        `timestamp` in the `extra` option to the logging call.
        """

        def filter(self, record):
            if hasattr(record, 'timestamp'):
                record.created = record.timestamp
            return True


# --- Standard initialization
handler = logging_loki.LokiHandler(
    url="http://git-lapp202.gtos.psa:3100/loki/api/v1/push",
    tags={"application": "my-app"},
    version="1",
)
logger = logging.getLogger("my-logger")
logger.addHandler(handler)

# --- Add timestampfilter ...
filter = TimestampFilter()
logger.addFilter(filter)


# --- Now log a record without time
logger.error( "Logging at actual time", extra={"tags": {"service": "my-service"}},)

# --- Now log a record 15 minutes ago
my_timestamp = time.time() -900
logger.error( "Logging in the past", extra={"tags": {"service": "my-service", 'timestamp' : my_timestamp}},)

Improve performance with batching

Currently it only uploads one log line with each HTTP request, which does not work well for high velocity logs.

The queued handler could potentially read all the available lines in the queue and upload everything outstanding at once as opposed to popping an item one at a time, and uploading it

duplicate lines being sent to loki

Hello

I am seeing that the LokiHandler gets called several times (up to 4 times) while the local logger only gets called once. Any idea why this could happen? Both with Queue and without

QueueHandler not working with LokiHandler

This code, gotten from the project examples, does not work. It does not produce any logs on Loki:

import logging.handlers
import logging_loki
from multiprocessing import Queue

queue = Queue(-1)

handler = logging.handlers.QueueHandler(queue)

handler_loki = logging_loki.LokiHandler(
    url="http://localhost:3100/loki/api/v1/push",
    tags={"app": "none", "job": "testing-loki", "env": "dev"},
    version="1",
)

logging.handlers.QueueListener(queue, handler_loki)

logger = logging.getLogger("my-logger")
logger.addHandler(handler)

if __name__ == '__main__':
    logger.info("Hello, world")
    logger.error("Something wrong...")

You may start Loki using Docker Compose docker-compose up -d in the same directory as the following docker-compose.ymlfile:

version: "3.7"
services:
  loki:
    image: grafana/loki:2.6.0
    ports:
      - "3100:3100"

Provide a PR for the original one / release this one?

As this fork is far more advanced then the base, could you please either create a PR for the base or make this one an official release on Pypi?

Thanks!

Cannot use log level "info" to send logs to loki

I am trying to push logs to loki.

When pushing logs with log level info , I do not see any logs going to loki .also it does not throw any error as well.
When i use log level as error/warn i can logs pushed to loki.

Do i need to make any change to send info log level logs to info loki?

logger.info(
    "test loki forward",
    extra={"tags": {"app": "app1"}},
)

Also i have used

from logging_loki import LokiHandler, emitter
emitter.LokiEmitter.level_tag = "level"

Errors when initializing

I am seeing some errors in the console. This happens erratically

Traceback (most recent call last):
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/site-packages/logging_loki/handlers.py", line 81, in emit
    self.emitter(record, self.format(record))
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/site-packages/logging_loki/emitter.py", line 57, in __call__
    raise ValueError("Unexpected Loki API response status code: {0}".format(resp.status_code))
ValueError: Unexpected Loki API response status code: 400
Call stack:
  File "/home/cristian/apps/pycharm-2019.2.2/plugins/python/helpers/pydev/_pydev_bundle/pydev_monkey.py", line 770, in __call__
    ret = self.original_func(*self.args, **self.kwargs)
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/logging/handlers.py", line 1479, in _monitor
    self.handle(record)
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/logging/handlers.py", line 1462, in handle
    handler.handle(record)
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/logging/__init__.py", line 865, in handle
    self.emit(record)
  File "/home/cristian/anaconda3/envs/actionserver/lib/python3.6/site-packages/logging_loki/handlers.py", line 83, in emit
    self.handleError(record)
Message: 'No retry needed.'

--- Logging error ---
Traceback (most recent call last):
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/site-packages/logging_loki/handlers.py", line 81, in emit
    self.emitter(record, self.format(record))
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/site-packages/logging_loki/emitter.py", line 57, in __call__
    raise ValueError("Unexpected Loki API response status code: {0}".format(resp.status_code))
ValueError: Unexpected Loki API response status code: 400
Call stack:
  File "/home/cristian/apps/pycharm-2019.2.2/plugins/python/helpers/pydev/_pydev_bundle/pydev_monkey.py", line 770, in __call__
    ret = self.original_func(*self.args, **self.kwargs)
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/logging/handlers.py", line 1479, in _monitor
    self.handle(record)
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/logging/handlers.py", line 1462, in handle
    handler.handle(record)
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/logging/__init__.py", line 865, in handle
    self.emit(record)
  File "/home/cristian/anaconda3/envs/chatbot/lib/python3.6/site-packages/logging_loki/handlers.py", line 83, in emit
    self.handleError(record)
Message: 'Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section'
Arguments: None

How to make fields?

Hello, I'm trying to make fields in grafana (as shown in the picture), but I don't understand how I should form a log sending from python to loki.

logger.info ('Test', extra={"tags": {"tag": 1}, "fields": {"field": 1}}) or...?

Use 'level' instead of 'severity'

Loki and Grafana use level as label to show the logging severity/verbose/levelname. It would be useful to replace the current severity label with level, as all queries and visualizations in Grafana would automatically pick up the log level.
I can create a pull request if you like, but let me know if the current has a special purpose.

Integrating with OpenTelemetry Python Logging

Hi,

Thanks for an awesome project, this is fantastically useful! :)

I'm trying to sort out my logging so that it sends the OpenTelemetry data such as trace ID's to Loki in order that I can look them up in Tempo.

At the moment, I have the following code:

import logging
from opentelemetry.instrumentation.logging import LoggingInstrumentor

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

import logging_loki


# Resource can be required for some backends, e.g. Jaeger
# If resource wouldn't be set - traces wouldn't appears in Jaeger
resource = Resource(attributes={
    "service.name": "test-otlp"
})

trace.set_tracer_provider(TracerProvider(resource=resource))
tracer = trace.get_tracer(__name__)
logger = logging.getLogger(__name__)
LoggingInstrumentor().instrument(set_logging_format=True)
handler = logging_loki.LokiHandler(
    url="https://loki/loki/api/v1/push", 
    tags={"application": "test-otlp"},
    version="1",
)

logger.addHandler(handler)

otlp_exporter = OTLPSpanExporter(endpoint="https://otlpgrpc")

span_processor = BatchSpanProcessor(otlp_exporter)

trace.get_tracer_provider().add_span_processor(span_processor)

with tracer.start_as_current_span("foo"):
    print("Hello world!")
    logger.info("Hello world!")

and this results in the following output on the console:

Hello world!
2021-07-10 08:11:49,314 INFO [__main__] [trace-otlp.py:39] [trace_id=f1d06cdda1d580e2003ba6227195f535 span_id=6222b514d274e0d2 resource.service.name=test-otlp] - Hello world!

However in Loki it shows up without the trace_id, span_id, or resource.servicename fields.

What am I missing here to get those fields into the Loki logs?

readme example needs to "start()" the listener

In the example demonstrating usage of QueueHandler and QueueListener, the listener is instantiated with the following line in the example:
logging.handlers.QueueListener(queue, handler_loki)

Looking at the source code, the thread handling the queue is not instantiated or started until start() is called (so __init__ is not enough).