GithubHelp home page GithubHelp logo

django-request-profiler's Introduction

Django Request Profiler

This package now requires Python 3.9 and Django 3.2 and above.

A very simple request profiler for Django.

Introduction

Premature optimization is the root of all evil.

There are a lot of very good, and complete, python and django profilers available. They can give you detailed stack traces and function call timings, output all the SQL statements that have been run, the templates that have been rendered, and the state of any / all variables along the way. These tools are great for optimisation of your application, once you have decided that the time is right.

django-request-profiler is not intended to help you optimise, but to help you decide whether you need to optimise in the first place. It is complimentary.

Requirements

  1. Small enough to run in production
  2. Able to configure profiling at runtime
  3. Configurable to target specific URLs or users
  4. Record basic request metadata:
  • Duration (request-response)
  • Request path, remote addr, user-agent
  • Response status code, content length
  • View function
  • Django user and session keys (if appropriate)
  • Database query count (if DEBUG=True)

It doesn't need to record all the inner timing information - the goal is to have a system that can be used to monitor site response times, and to identify problem areas ahead of time.

Technical details

The profiler itself runs as Django middleware, and it simply starts a timer when it first sees the request, and stops the timer when it is finished with the response. It should be installed as the first middleware in MIDDLEWARE_CLASSES in order to record the maximum duration.

It hooks into the process_request method to start the timer, the process_view method to record the view function name, and the process_response method to stop the timer, record all the request information and store the instance.

The profiler is controlled by adding RuleSet instances which are used to filter which requests are profiled. There can be many, overlapping, RuleSets, but if any match, the request is profiled. The RuleSet model defines two core matching methods:

1. uri_regex - in order to profile a subset of the site, you can supply a regex which is used match the incoming request path. If the url matches, the request can be profiled.

2. user_filter_type - there are three choices here - profile all users, profile only authenticated users, and profile authenticated users belonging to a given Group - e.g. create a groups called "profiling" and add anyone you want to profile.

These filter properties are an AND (must pass the uri and user filter), but the rules as a group are an OR - so if a request passes all the filters in any rule, then it's profiled.

These filters are pretty blunt, and there are plenty of use cases where you may want more sophisticated control over the profiling. There are two ways to do this. The first is a setting, REQUEST_PROFILER_GLOBAL_EXCLUDE_FUNC, which is a function that takes a request as the single argument, and must return True or False. If it returns False, the profile is cancelled, irrespective of any rules. The primary use case for this is to exclude common requests that you are not interested in, e.g. from search engine bots, or from Admin users etc. The default for this function is to prevent admin user requests from being profiled.

The second control is via the cancel() method on the ProfilingRecord, which is accessible via the request_profile_complete signal. By hooking in to this signal you can add additional processing, and optionally cancel the profiler. A typical use case for this is to log requests that have exceeded a set request duration threshold. In a high volume environment you may want to, for instance, only profile a random subset of all requests.

from django.dispatch import receiver
from request_profiler.signals import request_profile_complete

@receiver(request_profiler_complete)
def on_request_profile_complete(sender, **kwargs):
    profiler = kwargs.get('instance')
    if profiler.elapsed > 2:
        # log long-running requests
        # NB please don't use 'print' for real - use logging
        print u"Long-running request warning: %s" % profiler
    else:
        # calling cancel means that it won't be saved to the db
        profiler.cancel()

An additional scenario where you may want to use the signal is to store the profiler records async - say if you are recording every request for a short period, and you don't want to add unnecessary inline database write operations. In this case you can use the stop() method, which will prevent the middleware from saving it directly (it will only save records where profiler.is_running is true, and both cancel and stop set it to false).

from django.dispatch import receiver
from request_profiler.signals import request_profile_complete

@receiver(request_profiler_complete)
def on_request_profile_complete(sender, **kwargs):
    profiler = kwargs.get('instance')
    # stop the profiler to prevent it from being saved automatically
    profiler.stop()
    assert not profiler.is_running
    # add a job to a queue to perform the save itself
    queue.enqueue(profiler.save)

Installation

For use as the app in Django project, use pip:

$ pip install django-request-profiler
# For hacking on the project, pull from Git:
$ git pull [email protected]:yunojuno/django-request-profiler.git

Tests

The app installer contains a test suite that can be run using the Django test runner:

$ pip install -r requirements.txt
$ python manage.py test test_app request_profiler

If you want to test coverage you'll need to add some dependencies:

$ pip install coverage django-coverage
$ python manage.py test_coverage test_app request_profiler

The tests also run using tox:

$ pip install tox
$ tox

Note: To test with a custom user model, you should override the default User model by providing a value for the AUTH_USER_MODEL (in testapp/settings) setting that references a custom model

The tests run on Travis on commits to master.

Usage

Once installed, add the app and middleware to your project's settings file. In order to add the database tables, you should run the migrate command:

$ python manage.py migrate request_profiler

NB the middleware must be the first item in MIDDLEWARE_CLASSES.

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'request_profiler',
)

MIDDLEWARE_CLASSES = [
    # this package's middleware
    'request_profiler.middleware.ProfilingMiddleware',
    # default django middleware
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
]

Configuration

To configure the app, open the admin site, and add a new request profiler 'Rule set'. The default options will result in all non-admin requests being profiled.

Licence

MIT (see LICENCE)

django-request-profiler's People

Contributors

devharam avatar djm avatar emab avatar hugorodgerbrown avatar jamespeacock avatar miphreal avatar sigshen avatar wodow avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-request-profiler's Issues

Add query count to profiler record

When DEBUG=True Django will store a record of queries logged against a connection:

# django.db.backends.base.base.py#L149-L159
@property
def queries_logged(self):
    return self.force_debug_cursor or settings.DEBUG

@property
def queries(self):
    if len(self.queries_log) == self.queries_log.maxlen:
        warnings.warn(
            "Limit for query logging exceeded, only the last {} queries "
            "will be returned.".format(self.queries_log.maxlen))
    return list(self.queries_log)

The absolute number of queries executed during the course of a request is a very simple indicator of a possible issue. We should log it.

Timestamps are not tz-aware.

When running the profiler on a site that is timezone aware, am getting this all the time:

RuntimeWarning: DateTimeField ProfilingRecord.start_ts received a naive datetime (2014-10-17 13:48:36.380853) while time zone support is active.

RuntimeWarning: DateTimeField ProfilingRecord.end_ts received a naive datetime (2014-10-17 13:48:36.382230) while time zone support is active.

Need to ensure that we are using tz-aware datetimes wherever possible.

Support custom user models

$ ./manage.py check
CommandError: System check identified some issues:

ERRORS:
request_profiler.ProfilingRecord.user: (fields.E301) Field defines a relation with the model 'auth.User', which has been swapped out.
        HINT: Update the relation to point at 'settings.AUTH_USER_MODEL'.

Thats because you have swappable_dependency in your initial migration.

dependencies = [
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
    ]

Add support for custom filter functions

As well as regex URLs, would be nice / powerful to have custom functions that can evaluate a request and return true/false. Nothing complicated - just load them from Django settings.

I only see POST & OPTION

Hello,

I only see POST & OPTION requests

image

My filter is the default:

image

Do I do something wrong?

Note: I'm using the Django rest framework

Pip installation does't work.

$ pip install django-request—profiler
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 305, in run
    name, None, isolated=options.isolated_mode,
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 181, in from_line
    isolated=isolated)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 54, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2873, in parse
    reqs = list(parse_requirements(s))
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2820, in parse_requirements
    "version spec")
  File "/usr/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2785, in scan_list
    raise ValueError(msg, line, "at", line[p:])
ValueError: ('Expected version spec in', 'django-request\xe2\x80\x94profiler', 'at', '\xe2\x80\x94profiler')

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.