peopledoc / django-chunkator

Chunk large QuerySets into small chunks, and iterate over them without killing your RAM.

django-chunkator

Chunk large QuerySets into small chunks, and iterate over them without killing your RAM.

Tested with all the combinations of:

• Python: 3.5, 3.6, 3.7, 3.8
• Django: 2, 2.1, 2.2, 3.0, master

Note

Django 3.0 is incompatible with Python 3.5, see <https://docs.djangoproject.com/en/3.0/releases/3.0/#python-compatibility>

Usage

from chunkator import chunkator
for item in chunkator(LargeModel.objects.all(), 200):
    do_something(item)

This tool is intended to work on Django querysets.

Your model must define a pk field (this is done by default, but sometimes it can be overridden) and this pk has to be unique. django- chunkator has been tested with PostgreSQL and SQLite, using regular PKs and UUIDs as primary keys.

You can also use values():

from chunkator import chunkator
for item in chunkator(LargeModel.objects.values('pk', 'name'), 200):
    do_something(item)

Important

If you're using values() you have to add at least your "pk" field to the values, otherwise, the chunkator will throw a MissingPkFieldException.

Warning

This will not accelerate your process. Instead of having one BIG query, you'll have several small queries. This will save your RAM instead, because you'll not load a huge queryset result before looping on it.

If you want to manipulate the pages directly, you can use `chunkator_page`:

from chunkator import chunkator_page
queryset = LargeModel.objects.all().values('pk')
for page in chunkator_page(queryset, 200):
    launch_some_task([item['pk'] for item in page])

FAQ

• How is django-chunkator different from Django's iterator?

If you have server-side cursors (using Postgres or Oracle & not setting DISABLE_SERVER_SIDE_CURSORS), then the main difference is that the cursor is in the hands of the application instead of the server. It really depends on your constraints, but sometimes server side cursors can put too much strains on your DB.

If you don't have server-side cursors, then chunkator will allow you to iterate over your queryset by batch, without relying on LIMIT/OFFSET. The problem with LIMIT/OFFSET is that computing a large offset (when you're at the end of your queryset) requires the DB to go through all the previous entries. With large tables this can be a huge issue.

• Will django-chunkator preserve the ordering on my querysets?

No, it orders the queryset by pk. However you could do the same thing than chunkator with another field, given that it's unique and not nullable, see here for more details.

License

MIT License.

django-chunkator's Issues

Python 3.5 - PendingDeprecationWarning: generator 'chunkator_page' raised StopIteration

Hello!
I see this warning with python 3.5:

PendingDeprecationWarning: generator 'chunkator_page' raised StopIteration
  for page in chunkator_page(articles, chunk_size):

It's related to https://www.python.org/dev/peps/pep-0479/
which states StopIteration exception should no longer be raised manually inside generators, and replaced with a return statement.

How is django-chunkator different/better than `queryset.iterator()` ?

How is django-chunkator different / better than queryset.iterator ?

Confirm support for Django 3.0

Django version deprecations / upgrades

• deprecate django versions 1.6 and 1.7 gracefully,
• support django 1.9

FYI: works with Django 2.0.5

I've been using it with Django 2.0.5 without an issue. Just wanted to give a heads up that you can probably add this to the supported versions list.

Alternate sort

give an opportunity to sort by a number of criteria, and still loop over the records without losing one.

Drop `six` from requirements

This was necessary when using both Python2+3.

If queryset was ordered prior to calling `chunkator` would this ordering be preserved?

If queryset was ordered prior to calling chunkator would this ordering be preserved?

Python 3.9 support

title says it all

feature requests: support keep origin ordering of the queryset.

In most cases, We want to sort a queryset by update_datetime and then iterate it.

e.g. send a message when an new vedio was published or updated.

last_update_datetime = cache.get("last-update-datetime")
for vedio in chunkator(Vedio.objects.filter(update_datetime__gte=last_update_datetime).order_by("update_datetime", 200):
    send_notify(vedio)
    cache.set("last-update-datetime", vedio.update_datetime)

Python 3

using tox and travis.yml, let's make it Py3 compatible.

Compatibility with django-hashid-field

I am using django-hashid-field to obfuscate my pk fields.
It does not support gt operator, so source_qs.filter(pk__gt=pk) fails.
Is it an efficient alternative to do source_qs.all()[start:]? (need look to pk though...)
I opened an issue here: nshafer/django-hashid-field#38

Drop Python 2 support

[use a checklist]

Drop Python 3.5 support

ref: https://endoflife.date/python

Python 3.5 support was ended on September 2020.

Cleanup django support matrix

Some of them to be added, some of the to be removed (list TBD)

refs https://endoflife.date/django

Drop Django 1.11 support

Fix chunkator when the primary key is a OneToOneField

django-chunkator orders by pk.

But sometimes, you want to order by an explicit field.

For example:

class ModelA(models.Model):
    name = models.CharField(max_length=20)

    class Meta:
        ordering = ['name']


class ModelB(models.Model):
    modela = models.OneToOneField(ModelA, primary_key=True)

(with django 1.8 ?) when we write this queryset: ModelB.objects.order_by('pk'), it will in background order by modela__name.
We need to order the queryset by modela_id explicitely:

from chunkator import chunkator
for item in chunkator(ModelB.objects.all(), 200, order_by='modela_id'):
    do_something(item)

Edit: it is possible to fix this case without changing the API. See PR #20

Python 3 support in travis

update the travis.yml to build using python3