evgeneoskin / django-querysetsequence

Chain multiple (disparate) QuerySets in Django

Django QuerySetSequence

The QuerySetSequence wrapper helps to deal with disparate QuerySet classes, while treating them as a single QuerySet.

Supported Features

Listed below are features of Django's QuerySets_ that QuerySetSequence implements. The behavior should match that of QuerySet, but applied across multiple QuerySets:

• Methods that take a list of fields (e.g. filter(), exclude(), get(), order_by()) must use fields that are common across all sub-QuerySets.
• Relationships across related models work (e.g. 'foo__bar', 'foo', or 'foo_id'). syntax).
• The sub-QuerySets are evaluated as late as possible (e.g. during iteration, slicing, pickling, repr()/len()/list()/bool() calls).
• Public QuerySet API methods that are untested/unimplemented raise NotImplementedError. AttributeError is raised on attributes not explictly inherited from QuerySet.

QuerySet API implemented by QuerySetSequence

Method	Implemented?	Notes
filter()_	✓	See1 for information on the QuerySet lookup: '#'.
exclude()_	✓	See2 for information on the QuerySet lookup: '#'.
annotate()_	✗
order_by()_	✓	Does not support random ordering (e.g. order_by('?')). See3 for information on the QuerySet lookup: '#'.
reverse()_	✓
distinct()_	✗
values()_	✗
values_list()_	✗
dates()_	✗
datetimes()_	✗
none()_	✓
all()_	✓
select_related()_	✓
prefetch_related()_	✓
extra()_	✗
defer()_	✗
only()_	✗
using()_	✗
select_for_update()_	✗
raw()_	✗
get()_	✓	See4 for information on the QuerySet lookup: '#'.
create()_	✗	Cannot be implemented in QuerySetSequence.
get_or_create()_	✗	Cannot be implemented in QuerySetSequence.
update_or_create()_	✗	Cannot be implemented in QuerySetSequence.
bulk_create()_	✗	Cannot be implemented in QuerySetSequence.
count()_	✓
in_bulk()_	✗	Cannot be implemented in QuerySetSequence.
iterator()_	✓
latest()_	✗
earliest()_	✗
first()_	✗
last()_	✗
aggregate()_	✗
exists()_	✓
update()_	✗	Cannot be implemented in QuerySetSequence.
delete()_	✓
as_manager()_	✓

Requirements

• Python (2.7, 3.4, 3.5)
• Django (1.8, 1.9, 1.10)
• (Optionally) Django REST Framework (3.2, 3.3, 3.4)

QuerySetSequence versions with support for Django/Django REST Framework

	Django 1.8	Django 1.9	Django 1.10
Django REST Framework 3.2	0.7	✗	✗
Django REST Framework 3.3	0.7	0.7	✗
Django REST Framework 3.4	0.7	0.7	0.7

Installation

Install the package using pip.

pip install --upgrade django-querysetsequence

Usage

# Import QuerySetSequence
from queryset_sequence import QuerySetSequence

# Create QuerySets you want to chain.
from .models import SomeModel, OtherModel

# Chain them together.
query = QuerySetSequence(SomeModel.objects.all(), OtherModel.objects.all())

# Use query as if it were a QuerySet! E.g. in a ListView.

You can also provide a model keyword argument if you need to specify the QuerySet Model, e.g. for compatibility with some third-party applications that check the model field for equality

Example

class Author(models.Model):
    name = models.CharField(max_length=50)

    class Meta:
        ordering = ['name']

    def __str__(self):
        return self.name


class Article(models.Model):
    title = models.CharField(max_length=100)
    author = models.ForeignKey(Author)

    def __str__(self):
        return "%s by %s" % (self.title, self.author)


class Book(models.Model):
    title = models.CharField(max_length=50)
    author = models.ForeignKey(Author)
    release = models.DateField(auto_now_add=True)

    def __str__(self):
        return "%s by %s" % (self.title, self.author)

# Create some data.
alice = Author.objects.create(name='Alice')
article = Article.objects.create(title='Dancing with Django', author=alice)

bob = Author.objects.create(name='Bob')
article = Article.objects.create(title='Django-isms', author=bob)
article = Book.objects.create(title='Biography', author=bob)

# Create some QuerySets.
books = Book.objects.all()
articles = Article.objects.all()

# Combine them into a single iterable.
published_works = QuerySetSequence(books, articles)

# Find Bob's titles.
bob_works = published_works.filter(author=bob)
# Still an iterable.
print([w.title for w in bob_works])  # prints: ['Biography', 'Django-isms']

# Alphabetize the QuerySet.
published_works = published_works.order_by('title')
print([w.title for w in published_works])  # prints ['Biography', 'Dancing with Django', 'Django-isms']

Django REST Framework integration

django-querysetsequence comes with a custom CursorPagination class that helps integration with Django REST Framework. It is optimized to iterate over a QuerySetSequence first by QuerySet and then by the normal ordering configuration. This uses the optimized code-path for iteration that avoids interleaving the individual QuerySets. For example:

from queryset_sequence.pagination import SequenceCursorPagination

class PublicationPagination(SequenceCursorPagination):
    ordering = ['author', 'title']

class PublicationViewSet(viewsets.ModelViewSet):
    pagination_class = PublicationPagination

    def get_queryset(self):
        # This will return all Books first, then all Articles. Each of those
        # is individually ordered by ``author``, then ``title``.
        return QuerySetSequence(Book.objects.all(), Article.objects.all())

Attribution

This is based on a few DjangoSnippets that had been going around:

• Originally from https://www.djangosnippets.org/snippets/1103/
• Modified version from https://djangosnippets.org/snippets/1253/
• Upgraded version from https://djangosnippets.org/snippets/1933/
• Updated version from django-ko-demo from The Atlantic

Contribute

• Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
• Fork the repository on GitHub to start making your changes.
• Write a test which shows that the bug was fixed or that the feature works as expected.
• Send a pull request and bug the maintainer until it gets merged and published.

---

1. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩

2. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩

3. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩

4. QuerySetSequence supports a special field lookup that looks up the index of the QuerySet, this is represented by '#'. This can be used in any of the operations that normally take field lookups (i.e. filter(), exclude(), and get()), as well as order_by().

   A few examples are below:

   # Order first by QuerySet, then by the value of the 'title' field.
   QuerySetSequence(...).order_by('#', 'title')
   
   # Filter out the first QuerySet.
   QuerySetSequence(...).filter(**{'#__gt': 0})

   Note

   Ordering first by QuerySet allows for a more optimized code path when iterating over the entries.

   Warning

   Not all lookups are supported when using '#' (some lookups simply don't make sense; others are just not supported). The following are allowed:

   • exact
   • iexact
   • contains
   • icontains
   • in
   • gt
   • gte
   • lt
   • lte
   • startswith
   • istartswith
   • endswith
   • iendswith
   • range
   ↩