dabapps / django-log-request-id Goto Github PK

View Code? Open in Web Editor NEW

364.0 364.0 63.0 105 KB

Django middleware and log filter to attach a unique ID to every log message generated as part of a request

License: BSD 2-Clause "Simplified" License

Python 100.00%

django-log-request-id's People

Contributors

Stargazers

Watchers

django-log-request-id's Issues

Log request ip address

It would be helpful to log the ip address from the request

Replacing the text `request_id` to `correlation_id`

According to the current setup we are getting the log like this
{"request_id": "49479a875c5b45439bf4e480efd7b0ec", "asctime": "2022-01-10 09:00:38,878", .....
Is there any way to customise the text like
{"correlation_id": "49479a875c5b45439bf4e480efd7b0ec", "asctime": "2022-01-10 09:00:38,878",

Library leads to Vary: Cookie header set

Greetings,

thank you for the lovely package!
After checking current get_log_message, I am seeing that we will always try to grab a user attribute (and fall back to user.pk or user.id if no custom attribute was set).

And there is an interesting side effect coming from that fact:

request.user is a lazy object until we start touching it
the moment we access it, Django uses the session to verify that user is associated or returns AnonymousUser instance
the moment we access a session, SessionMiddleware sets vary cookie to the headers

This logic means that even if we serve some simple static request that is not doing anything with the user instance - django-log-request-id middleware will still lead to vary cookie be set.

So, in short, that means, that logging library is changing response headers, which is an arguable behaviour.

I'd love to get your opinion on this first and then, if valid, continue discussion with possible mitigation options and improvements we could make.

Best,
Rust

Can not install version 1.4.0 on python3.7.4 (python3.6, no problem)

(venv37) tj001@Ubantu18:~/code/lengine-service$ python -m pip install django-log-request-id

Collecting django-log-request-id
  Using cached https://files.pythonhosted.org/packages/17/2f/d283eacfc9a149f65298eb0a4eec6d85a9589e15e9f11468f69ab183a250/django-log-request-id-1.4.0.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /home/tj001/env/venv37/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8d592g5r/django-log-request-id/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8d592g5r/django-log-request-id/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-8d592g5r/django-log-request-id/pip-egg-info
         cwd: /tmp/pip-install-8d592g5r/django-log-request-id/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/tj001/env/venv37/lib/python3.7/site-packages/setuptools/__init__.py", line 20, in <module>
        from setuptools.dist import Distribution, Feature
      File "/home/tj001/env/venv37/lib/python3.7/site-packages/setuptools/dist.py", line 36, in <module>
        from setuptools import windows_support
      File "/home/tj001/env/venv37/lib/python3.7/site-packages/setuptools/windows_support.py", line 2, in <module>
        import ctypes
      File "/usr/local/lib/python3.7/ctypes/__init__.py", line 7, in <module>
        from _ctypes import Union, Structure, Array
    ModuleNotFoundError: No module named '_ctypes'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Add Official Support for Python 3.6 and Django 1.11

To use this library with newer django projects, I would like official support for python 2.6 and Django 1.11
I would also like to fix the Travis build to use Python 3.6.
- Python 3.5 isn't a useful or common version to test on, and isn't available in the Travis build
The tests use a deprecated urlpatterns tool which I would like to only rely on for older versions of Django
- This tool was removed in Django 1.11, deprecated in Django 1.8

PR: #25

request_id in celery logs

Is there any way of getting the request_id from within celery tasks?

Does it support new-style middleware?

Hi,
I noticed that the package uses MIDDLEWARE_CLASSES, which is the old style middleware. Does it support also new style?
Thanks.

Heroku request timeout and status=200

I'm using django-log-request-id with Heroku, and I noticed in the logs that when I get H12 request timeouts from Heroku for any request (due to 30 second time limit), django-log-request-id registers it with status=200. The expected behavior should be to use status=503, which is what Heroku responds with. Here is an example:

Feb 25 09:33:19 my-app heroku/router:  at=error code=H12 desc="Request timeout" method=GET path="/my-url" host=my.host.com request_id=dad4ad83-0a55-447d-be83-28d6ea38bc55 dyno=web.1 connect=25ms service=30005ms status=503 bytes=0 
Feb 25 09:34:09 my-app app/web.1:  dad4ad83-0a55-447d-be83-28d6ea38bc55 [INFO] [middleware.process_response(36)]: method=GET path=/my-url status=200 user=4446

Log request id set to [none] in thread pool executor.

In the thread pull executor, I am calling multiple 3rd party APIs in parallel but in the log, it set the log request id none.

ERROR    [none] [2022-01-19 12:57:16,200] weather.apis:  for attempt : 1
ERROR    [none] [2022-01-19 12:57:16,220] weather.apis:  for attempt : 2
ERROR    [none] [2022-01-19 12:57:16,239] weather.apis:  for attempt : 3
ERROR    [none] [2022-01-19 12:57:16,239] weather.apis: Vendor API failed.

Link for thread pool executor:
https://docs.python.org/3/library/concurrent.futures.html

Generate/User Log Request id based on header.

Create a new log request id if not passed into the header, otherwise use the log_request_id that passed into the header for log.

In my case I am using the package in microservices to get the same log request id but how can I ensure that it does not generate the new log request id if the endpoint is called from another service and use the same one passed into the header, or ensure that if a new request is created by the user there will be no log request-id and we need to generate a new id.

Access to the Id

Hey Guys,

Firstly this is a great project, I can't believe this isn't baked into Django already.

not even sure that this is an issue but I would like to access the ID that is generated so that i can put it in a Json response so that users can give this to the team for tshooting, how can I access this from the view perspective ?

Thanks

Can not install version 1.4.0 on python 3

Issue installing with pip on python 3.6.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-st95698j/django-log-request-id/setup.py", line 23, in <module>
    readme = f.read()
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 5406: ordinal not in range(128)

No documentation to add filter to custom logger

I implemented a custom logger which I am using in my Django project

import logging
import sys


def setup_custom_logger(name):
    logger = logging.getLogger(name)
    # logger.addFilter(reques_)
    formatter = logging.Formatter(fmt='%(asctime)s,%(msecs)d %(levelname)-3s [%(filename)s:%(lineno)d] %(message)s',
                                  datefmt='%Y-%m-%d %H:%M:%S')

    screen_handler = logging.StreamHandler(stream=sys.stdout)
    screen_handler.setFormatter(formatter)
    logger.setLevel(logging.INFO)
    logger.addHandler(screen_handler)
    return logger

This is used as

 logger = setup_custom_logger(__name__)

And I cannot find anywhere in the documentation of how to add the filter to the given logger object

Thoughts abt how this works when ASGI rolls out in 3.2?

Add processing_time to log line

Would you be open to adding a processing_time metric to each log line? It would keep track of the time from process_request to process_response. I can implement it and send you a pull request.

path with query string

Hi,

First of all, I appreciate this package.
I'd like to try this package in my project, so I wonder if you could help me out.

I hope the path of the message is also displayed as a query string.
I mean, I hope request.get_full_path() is used instead of request.path in the get_log_message function located in your RequestIDMiddleware class.
request.path is very inconvenient to know the exact url because it cuts all the paths behind '/?'.

Would that be possible?

Allow use of user_id if available

Thanks for building this repo!

I was wondering if a setting for something like USE_USER_ID_AS_REQUEST_ID: bool = False would be something you'd consider adding?

What I'm looking to do is have the unique id be the user's id/pk if that is available on the request object, otherwise fallback to the generated uuid.

A thought would be to prefix an actual user id with something like f"user-{user_id}" to differentiate it from the regular uuid.

Another solution would be to include an optional user_id setting on the formatter.

Let me know what you think? Happy to create a PR if it's welcome!

Security Concerns

First of all, this tool is so useful for me. I appreciate too much your effort.

I'm using your package and I see there is no limitation in the length of the X-Request-Id. I mean I can send thousands of characters as id and that could affect the size of my logs. Could you establish a limit? I mean you could truncate the string if the length is major than the limit expected.

Also, it would be good if the name of the header X-Request-Id can be customized.

Thank you in advance.

Unable to configure filter 'request_id': 'module' object has no attribute 'middleware

logging filters 'request_id' is app.middleware.RequestIDFilter

Question: how is it better then simple process/thread ID logging

Let's imagine that i don't use X_REQUEST_ID and LOG_REQUESTS = True features.
How this application is more useful than simple process/thread ID logging?
May be there some issues that I'm not aware of.

Example of simple thread/process ID logging:

'formatters': {
        'verbose': {
            'format': '%(levelname)s | %(asctime)s | %(name)s | %(process)d | %(thread)d | %(message)s'
        },
    },

Thank you.

Delete the request ID from the thread local in process_response

To prevent leakage (shouldn't happen anyway, but just to be safe)

1.5.0 break the compatibility

I got this error for the new version of log-request-id
from log_request_id import local, NO_REQUEST_ID, OUTGOING_REQUEST_ID_HEADER_SETTING ImportError: cannot import name NO_REQUEST_ID

Copyright outdated

Thank you for this library.

BTW, the Copyright in the README is outdated:

Copyright © 2012-2013, DabApps.

KeyError: 'request_id'

Hi, i followed strictly the readme on how to install, and i found this error:

--- Logging error ---
Traceback (most recent call last):
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/logging/__init__.py", line 1034, in emit
    msg = self.format(record)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/logging/__init__.py", line 880, in format
    return fmt.format(record)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/logging/__init__.py", line 622, in format
    s = self.formatMessage(record)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/logging/__init__.py", line 591, in formatMessage
    return self._style.format(record)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/logging/__init__.py", line 433, in format
    return self._fmt % record.__dict__
KeyError: 'request_id'
Call stack:
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/threading.py", line 885, in _bootstrap
    self._bootstrap_inner()
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/socketserver.py", line 720, in __init__
    self.handle()
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/servers/basehttp.py", line 154, in handle
    handler.run(self.server.get_app())
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/contrib/staticfiles/handlers.py", line 66, in __call__
    return self.application(environ, start_response)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 146, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/base.py", line 81, in get_response
    response = self._middleware_chain(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/deprecation.py", line 95, in __call__
    response = self.get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/decorators.py", line 62, in _wrapper
    return bound_func(*args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/decorators.py", line 58, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/decorators.py", line 62, in _wrapper
    return bound_func(*args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/decorators.py", line 142, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/decorators.py", line 58, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/decorators.py", line 62, in _wrapper
    return bound_func(*args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/utils/decorators.py", line 58, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/contrib/auth/views.py", line 65, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/views/generic/base.py", line 89, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/moonblade/devel/python/miniconda3/lib/python3.7/site-packages/django/views/generic/edit.py", line 133, in get
    return self.render_to_response(self.get_context_data())
  File "/home/moonblade/devel/java/eclipse_workspace/Daruma/daruma/apps/authentication/views.py", line 193, in get_context_data
    logger.debug("logger CCTV Started in login process!")
Message: 'logger CCTV Started in login process!'
Arguments: ()

I wonder what i missed ?
Please help.

Add documentation how to access the request ID in views

When integrating the middleware, I first tried to access the request ID in views through the headers array in the request request.headers["X-Request-ID"]. It would be helpful if this were documented in the readme instead of having to look at the source code to see that it is saved as request.id.

Is this already documented somewhere or should I create a pull request (that also extends the example)?

Documentation does not specify where to define options/config settings?

For example, regarding the auto-generation of a request id if there is none found in the headers the documentation says the following:

If you wish to fall back to a generated ID when you have the LOG_REQUEST_ID_HEADER set but it was not provided in the request, add the following setting:
GENERATE_REQUEST_ID_IF_NOT_IN_HEADER = True

Where should these variables be defined? Are they to be set in settings.py? I've tried adding them there, for both the GENERATE_REQUEST_ID_IF_NOT_IN_HEADER option and the NO_REQUEST_ID option, but they do not seem to be working. Any help would be appreciated

Doesn't log request id when you have multiple threads in the application.

When the Django app creates multiple threads using threading.Thread to handle the incoming requests, it fails to log the request id and gives the following error:

AttributeError: 'thread._local' object has no attribute 'request_id'

doesn't work with gevent

If I am not mistaken, the threading.local is based on thread id to store data, but django + gunicorn + gevent may only one thread.

Request ID is None

I am using Django2.1.3 and I have followed the exact same steps as described in the Readme.However, the request ID field comes as None as follows:

INFO     [2018-11-15 15:44:55,818] [none] connectivity_service.workflows.IPGroupCreateWorkFlow: {'status': 'IP Group created successfully on Junos Space'}

What could be wrong ??

[Question] is it a common use case to use this with sentry?

For production environments I mean

Would be great if there's documentation about using this with sentry if it's indeed a valid and common use case

Exception "You cannot access body after reading from request's data stream"

Hi, I'm seeing in my pyrest test that after making django-log-request-id the top of the middelware list, Django Rest Framework complaints that something already accessed the body

Exception trying to access request body: You cannot access body after reading from request's data stream

I imagine that's because we already read the headers of the request to verify the REQUEST-ID header or because we are adding therequest.id value?

has anyone experience this problem?

1.1.0 distribution on pypi contains pycache folder for cpython-34

Downloading django-log-request-id-1.1.0.tar.gz from https://pypi.python.org/pypi/django-log-request-id/1.1.0 contains a __pycache__ folder of pyc files for cpython-34.
That probably shouldn't be there.
I think it is the reason I'm getting ValueError: bad marshal data (unknown type code) when trying to build on python 2.7.

dabapps / django-log-request-id Goto Github PK

django-log-request-id's People

Contributors

Stargazers

Watchers

Forkers

django-log-request-id's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs