stefanfoulis / django-graceful-session-cleanup

A simple management command that can delete expired sessions from large session tables without killing the site.

django-graceful-session-cleanup

Warning: currently only works with mysql

A simple management command that can delete expired sessions from large session tables without killing the site.

The main use case is if you have a live application with millions of stale sessions in the database. Runnung the regular python manage.py cleanup will potentially lock your database for hours, essentially killing your site.

With django-graceful-session-cleanup you can gradually delete all the sessions.

To achieve this it will always only delete a small amount of the expired sessions per query and then sleep for a few seconds. There are commandline options to control interval, sleep time and amount of sessions to delete per interval.

Installation

install the package:

pip install django-graceful-session-cleanup

add it to INSTALLED_APPS:

INSTALLED_APPS = [
    ....
    'graceful_session_cleanup',
]

run the management command:

python manage.py graceful_session_cleanup

There are a few options you can use:

python manage.py graceful_session_cleanup --sleep-seconds 9 --delete-count 1000 --iteration-count 200

This will delete 1000 expired session entries, wait for 5 seconds so other processes can use the database and then repeat this 200 times. Depending on database load (assuming deleting takes 1s) this will take (9s + 1s) * 200 = 2000s.

there is help on the commandline:

$ python manage.py help graceful_session_cleanup
Usage: django graceful_session_cleanup [options]

Can be run as a cronjob or directly to clean out old data from the database (only expired sessions at the moment). Does this in a live db friendly way by never hogging the connection too long.

Options:
  -v VERBOSITY, --verbosity=VERBOSITY
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=all output
  --settings=SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath=PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Print traceback on exception
  -s SLEEP_SECONDS, --sleep-seconds=SLEEP_SECONDS
                        How long to sleep between each delete operation.
  -c DELETE_COUNT, --delete-count=DELETE_COUNT
                        How many expired sessions to delete per iteration.
  -i ITERATION_COUNT, --iteration-count=ITERATION_COUNT
                        How many iterations to run.
  --version             show program's version number and exit
  -h, --help            show this help message and exit

django-graceful-session-cleanup's Issues

add a "do it forever" option

make the command runnable by supervisord or similar in an "endless" mode. running it from crontab adds the risk that multiple running processes may pile up.

Doesn't support PostgreSQL (or mention supported DBs)

PostgreSQL doesn't support LIMIT on a DELETE FROM statement, so this management command fails.

At a minimum the README should mention that django-graceful-session-cleanup is currently only for MySQL.

investigate closing database connection between iterations

will this help at all?

Provide command to optimize table

OPTIMIZE TABLE django_session;

does this work on other db backends?

provide command to show session count

add a command or option to display the amount of sessions in the database.

• Session.objects.all().count() seems to work really fast, even with huge tables (tested on mysql)
• Session.objects.filter(expire_date__lt=datetime.datetime.now()).count() took over 2 minutes on my test database with 22M records and 11M expired sessions. -> don't provide this or add a huge warning.

optimize query for low numbers of expired sessions

The less expired sessions are in the database, the longer the delete query takes. This is probably because the database backend has to go thorough the whole list of not expired sessions desperately looking for ones to delete.

Currently it takes 8s to delete just 40 Sessions with a table with 11M active Sessions on our testsystem. Probably this would be a good case to just use Redis for sessions.

transaction.commit_unless_managed() missing

It seems that a call to transaction.commit_unless_managed() is missing after the cursor.execute() call. Without this, for me, running Mysql with InnoDB as the database engine, nothing actually ends up getting deleted (even though the log output from graceful session cleanup reports that rows are being deleted).

Does this sound right? It seems like something obvious that would have been noticed by now.