celery / django-celery-results

Celery result back end with django

Version

5.4.0 (opalescent)

Web

https://docs.celeryq.dev/en/stable/index.html

Download

https://pypi.org/project/celery/

Source

https://github.com/celery/celery/

Keywords

task, queue, job, async, rabbitmq, amqp, redis, python, distributed, actors

Donations

This project relies on your generous donations.

If you are using Celery to create a commercial product, please consider becoming our backer or our sponsor to ensure Celery's future.

For enterprise

Available as part of the Tidelift Subscription.

The maintainers of celery and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source dependencies you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact dependencies you use. Learn more.

What's a Task Queue?

Task queues are used as a mechanism to distribute work across threads or machines.

A task queue's input is a unit of work, called a task, dedicated worker processes then constantly monitor the queue for new work to perform.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker.

A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.

Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there's node-celery for Node.js, a PHP client, gocelery, gopher-celery for Go, and rusty-celery for Rust.

Language interoperability can also be achieved by using webhooks in such a way that the client enqueues an URL to be requested by a worker.

What do I need?

Celery version 5.3.5 runs on:

• Python (3.8, 3.9, 3.10, 3.11, 3.12)
• PyPy3.9+ (v7.3.12+)

This is the version of celery which will support Python 3.8 or newer.

If you're running an older version of Python, you need to be running an older version of Celery:

• Python 3.7: Celery 5.2 or earlier.
• Python 3.6: Celery 5.1 or earlier.
• Python 2.7: Celery 4.x series.
• Python 2.6: Celery series 3.1 or earlier.
• Python 2.5: Celery series 3.0 or earlier.
• Python 2.4: Celery series 2.2 or earlier.

Celery is a project with minimal funding, so we don't support Microsoft Windows but it should be working. Please don't open any issues related to that platform.

Celery is usually used with a message broker to send and receive messages. The RabbitMQ, Redis transports are feature complete, but there's also experimental support for a myriad of other solutions, including using SQLite for local development.

Celery can run on a single machine, on multiple machines, or even across datacenters.

Get Started

If this is the first time you're trying to use Celery, or you're new to Celery v5.4.x coming from previous versions then you should read our getting started tutorials:

• First steps with Celery

  Tutorial teaching you the bare minimum needed to get started with Celery.

• Next steps

  A more complete overview, showing more features.

You can also get started with Celery by using a hosted broker transport CloudAMQP. The largest hosting provider of RabbitMQ is a proud sponsor of Celery.

Celery is...

• Simple

  Celery is easy to use and maintain, and does not need configuration files.

  It has an active, friendly community you can talk to for support, like at our mailing-list, or the IRC channel.

  Here's one of the simplest applications you can make:

  from celery import Celery
  
  app = Celery('hello', broker='amqp://guest@localhost//')
  
  @app.task
  def hello():
      return 'hello world'

• Highly Available

  Workers and clients will automatically retry in the event of connection loss or failure, and some brokers support HA in way of Primary/Primary or Primary/Replica replication.

• Fast

  A single Celery process can process millions of tasks a minute, with sub-millisecond round-trip latency (using RabbitMQ, py-librabbitmq, and optimized settings).

• Flexible

  Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, broker transports, and much more.

It supports...

• Message Transports

  • RabbitMQ, Redis, Amazon SQS

• Concurrency

  • Prefork, Eventlet, gevent, single threaded (solo)

• Result Stores

  • AMQP, Redis
  • memcached
  • SQLAlchemy, Django ORM
  • Apache Cassandra, IronCache, Elasticsearch

• Serialization

  • pickle, json, yaml, msgpack.
  • zlib, bzip2 compression.
  • Cryptographic message signing.

Framework Integration

Celery is easy to integrate with web frameworks, some of which even have integration packages:

Django	not needed
Pyramid	pyramid_celery
Pylons	celery-pylons
Flask	not needed
web2py	web2py-celery
Tornado	tornado-celery
FastAPI	not needed

The integration packages aren't strictly necessary, but they can make development easier, and sometimes they add important hooks like closing database connections at fork.

Documentation

The latest documentation is hosted at Read The Docs, containing user guides, tutorials, and an API reference.

最新的中文文档托管在 https://www.celerycn.io/ 中，包含用户指南、教程、API接口等。

Installation

You can install Celery either via the Python Package Index (PyPI) or from source.

To install using pip:

:

$ pip install -U Celery

Bundles

Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature.

You can specify these in your requirements or on the pip command-line by using brackets. Multiple bundles can be specified by separating them by commas.

:

$ pip install "celery[redis]"

$ pip install "celery[redis,auth,msgpack]"

The following bundles are available:

Serializers

celery[auth]

for using the auth security serializer.

celery[msgpack]

for using the msgpack serializer.

celery[yaml]

for using the yaml serializer.

Concurrency

celery[eventlet]

for using the eventlet pool.

celery[gevent]

for using the gevent pool.

Transports and Backends

celery[amqp]

for using the RabbitMQ amqp python library.

celery[redis]

for using Redis as a message transport or as a result backend.

celery[sqs]

for using Amazon SQS as a message transport.

celery[tblib]

for using the task_remote_tracebacks feature.

celery[memcache]

for using Memcached as a result backend (using pylibmc)

celery[pymemcache]

for using Memcached as a result backend (pure-Python implementation).

celery[cassandra]

for using Apache Cassandra/Astra DB as a result backend with the DataStax driver.

celery[azureblockblob]

for using Azure Storage as a result backend (using azure-storage)

celery[s3]

for using S3 Storage as a result backend.

celery[gcs]

for using Google Cloud Storage as a result backend.

celery[couchbase]

for using Couchbase as a result backend.

celery[arangodb]

for using ArangoDB as a result backend.

celery[elasticsearch]

for using Elasticsearch as a result backend.

celery[riak]

for using Riak as a result backend.

celery[cosmosdbsql]

for using Azure Cosmos DB as a result backend (using pydocumentdb)

celery[zookeeper]

for using Zookeeper as a message transport.

celery[sqlalchemy]

for using SQLAlchemy as a result backend (supported).

celery[pyro]

for using the Pyro4 message transport (experimental).

celery[slmq]

for using the SoftLayer Message Queue transport (experimental).

celery[consul]

for using the Consul.io Key/Value store as a message transport or result backend (experimental).

celery[django]

specifies the lowest version possible for Django support.

You should probably not use this in your requirements, it's here for informational purposes only.

Downloading and installing from source

Download the latest version of Celery from PyPI:

https://pypi.org/project/celery/

You can install it by doing the following:

:

$ tar xvfz celery-0.0.0.tar.gz
$ cd celery-0.0.0
$ python setup.py build
# python setup.py install

The last command must be executed as a privileged user if you aren't currently using a virtualenv.

Using the development version

With pip

The Celery development version also requires the development versions of kombu, amqp, billiard, and vine.

You can install the latest snapshot of these using the following pip commands:

:

$ pip install https://github.com/celery/celery/zipball/main#egg=celery
$ pip install https://github.com/celery/billiard/zipball/main#egg=billiard
$ pip install https://github.com/celery/py-amqp/zipball/main#egg=amqp
$ pip install https://github.com/celery/kombu/zipball/main#egg=kombu
$ pip install https://github.com/celery/vine/zipball/main#egg=vine

With git

Please see the Contributing section.

Getting Help

Mailing list

For discussions about the usage, development, and future of Celery, please join the celery-users mailing list.

IRC

Come chat with us on IRC. The #celery channel is located at the Libera Chat network.

Bug tracker

If you have any suggestions, bug reports, or annoyances please report them to our issue tracker at https://github.com/celery/celery/issues/

Wiki

https://github.com/celery/celery/wiki

Credits

Contributors

This project exists thanks to all the people who contribute. Development of celery happens at GitHub: https://github.com/celery/celery

You're highly encouraged to participate in the development of celery. If you don't like GitHub (for some reason) you're welcome to send regular patches.

Be sure to also read the Contributing to Celery section in the documentation.

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

This software is licensed under the New BSD License. See the LICENSE file in the top distribution directory for the full license text.

django-celery-results's Issues

Cannot save GroupResult

I want to run multiple jobs as a group and want to store the result to watch the progress of the combined job.
According to the documentation I can use GroupResult.save and restore it later.

When I try to do that I get the following error message:

ERROR	2017-11-10 09:05:26,293 exception 7586 139766003652352 Internal Server Error:
Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "venv/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "venv/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "venv/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "venv/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "venv/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", line 78, in post
    async_result.save()
  File "venv/lib/python3.6/site-packages/celery/result.py", line 833, in save
    return (backend or self.app.backend).save_group(self.id, self)
  File "venv/lib/python3.6/site-packages/celery/backends/base.py", line 386, in save_group
    return self._save_group(group_id, result)
AttributeError: 'DatabaseBackend' object has no attribute '_save_group'

This makes sense as django_celery_results.backends.database.DatabaseBackend does not implement this method. celery.backends.database.DatabaseBackend Implements it but it is not used when set CELERY_RESULT_BACKEND = 'django-db'.

Did I configure this incorrectly or is this simply not implemented?

What I want to do is return the task_id of the group when a combined job is submitted and then poll the status of the group to see when all tasks are finished and not check every sub-task individually.

pip does not install migrations

When doing pip install django-celery-results, the migrations directory is not installed. The consequence is that doing python manage.py migrate does not create the results table. Doing the pip install directly from GitHub works fine.

TaskResult doesn't send Django post_save signals

I'm trying to do something when a task being written to database or its status updates with code like this:

from django.db.models.signals import post_save
from django_celery_results.models import TaskResult
from django.dispatch import receiver

@receiver(post_save, sender=TaskResult, dispatch_uid='task_update') 
def task_update(sender, instance, created, **kwargs):
    # do something

But nothing happened. Did I miss something, or I should use celery.signals?

I'm using rabbit-mq, and some packages' versions are following:

Django (1.11.4)
django-celery-results (1.0.1)
celery (4.1.0)

Hardcoded django-cache backend serializer

Is there a reason why the cache backend hardcodes its serializer to pickle?

https://github.com/celery/django-celery-results/blob/master/django_celery_results/backends/cache.py#L16

I'm using this to cache results in redis, since django allows me to configure redis SSL support via the rediss connection string and I don't have another option in production with Heroku. Using my redis SSL connection string as the backend breaks.

Going this route is working, but now I'm stuck with the pickle serializer. Looking for a way to help improve support for this case and trying to understand first!

OperationalError: (2006, 'MySQL server has gone away') after upgrading from celery 4.1.0 to 4.2.0

I'm using an amqp broker and the following packages:

Django==1.11.14
django-celery-results==1.0.1
amqp==2.3.2
kombu==4.2.1

After upgrading from celery 4.1.0 to 4.2.0, I am getting the following error:

[2018-07-18 18:47:02,439: ERROR/ForkPoolWorker-1] Task my_app.common.tasks.echo[f8e242bf-14af-48c7-a20f-9d6d94e4cc45] raised unexpected: OperationalError(2006, 'MySQL server has gone away')
Traceback (most recent call last):
  File "/srv/my_app/local/lib/python2.7/site-packages/celery/app/trace.py", line 442, in trace_task
    uuid, retval, task_request, publish_result,
  File "/srv/my_app/local/lib/python2.7/site-packages/celery/backends/base.py", line 146, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/srv/my_app/local/lib/python2.7/site-packages/celery/backends/base.py", line 322, in store_result
    request=request, **kwargs)
  File "/srv/my_app/local/lib/python2.7/site-packages/django_celery_results/backends/database.py", line 29, in _store_result
    meta=meta,
  File "/srv/my_app/local/lib/python2.7/site-packages/django_celery_results/managers.py", line 50, in _inner
    return fun(*args, **kwargs)
  File "/srv/my_app/local/lib/python2.7/site-packages/django_celery_results/managers.py", line 119, in store_result
    obj, created = self.get_or_create(task_id=task_id, defaults=fields)
  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 464, in get_or_create
    return self.get(**lookup), False
  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get
    num = len(clone)
  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__
    self._fetch_all()
  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
OperationalError: (2006, 'MySQL server has gone away')

The strange part is the task result actually makes it to the database.

In [3]: vars(TaskResult.objects.get(task_id='f8e242bf-14af-48c7-a20f-9d6d94e4cc45'))
lims.settings django.db.backends DEBUG (0.002) SELECT `django_celery_results_taskresult`.`id`, `django_celery_results_taskresult`.`task_id`, `django_celery_results_taskresult`.`status`, `django_celery_results_taskresult`.`content_type`, `django_celery_results_taskresult`.`content_encoding`, `django_celery_results_taskresult`.`result`, `django_celery_results_taskresult`.`date_done`, `django_celery_results_taskresult`.`traceback`, `django_celery_results_taskresult`.`hidden`, `django_celery_results_taskresult`.`meta` FROM `django_celery_results_taskresult` WHERE `django_celery_results_taskresult`.`task_id` = 'f8e242bf-14af-48c7-a20f-9d6d94e4cc45'; args=('f8e242bf-14af-48c7-a20f-9d6d94e4cc45',)
lims.settings django.db.backends DEBUG (0.002) SELECT `django_celery_results_taskresult`.`id`, `django_celery_results_taskresult`.`task_id`, `django_celery_results_taskresult`.`status`, `django_celery_results_taskresult`.`content_type`, `django_celery_results_taskresult`.`content_encoding`, `django_celery_results_taskresult`.`result`, `django_celery_results_taskresult`.`date_done`, `django_celery_results_taskresult`.`traceback`, `django_celery_results_taskresult`.`hidden`, `django_celery_results_taskresult`.`meta` FROM `django_celery_results_taskresult` WHERE `django_celery_results_taskresult`.`task_id` = 'f8e242bf-14af-48c7-a20f-9d6d94e4cc45'; args=('f8e242bf-14af-48c7-a20f-9d6d94e4cc45',)
Out[3]:
{'_state': <django.db.models.base.ModelState at 0x7fc6f86acfd0>,
 'content_encoding': u'utf-8',
 'content_type': u'application/json',
 'date_done': datetime.datetime(2018, 7, 18, 18, 47, 2),
 'hidden': False,
 'id': 7L,
 'meta': u'{"children": []}',
 'result': u'{"exc_message": [2006, "MySQL server has gone away"], "exc_module": "django.db.utils", "exc_type": "OperationalError"}',
 'status': u'FAILURE',
 'task_id': u'f8e242bf-14af-48c7-a20f-9d6d94e4cc45',
 'traceback': u'Traceback (most recent call last):\n  File "/srv/my_app/local/lib/python2.7/site-packages/celery/app/trace.py", line 442, in trace_task\n    uuid, retval, task_request, publish_result,\n  File "/srv/my_app/local/lib/python2.7/site-packages/celery/backends/base.py", line 146, in mark_as_done\n    self.store_result(task_id, result, state, request=request)\n  File "/srv/my_app/local/lib/python2.7/site-packages/celery/backends/base.py", line 322, in store_result\n    request=request, **kwargs)\n  File "/srv/my_app/local/lib/python2.7/site-packages/django_celery_results/backends/database.py", line 29, in _store_result\n    meta=meta,\n  File "/srv/my_app/local/lib/python2.7/site-packages/django_celery_results/managers.py", line 50, in _inner\n    return fun(*args, **kwargs)\n  File "/srv/my_app/local/lib/python2.7/site-packages/django_celery_results/managers.py", line 119, in store_result\n    obj, created = self.get_or_create(task_id=task_id, defaults=fields)\n  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 464, in get_or_create\n    return self.get(**lookup), False\n  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get\n    num = len(clone)\n  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__\n    self._fetch_all()\n  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__\n    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)\n  File "/srv/my_app/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql\n    raise original_exception\nOperationalError: (2006, \'MySQL server has gone away\')\n'}

If I switch to an rpc result backend instead of django-db, the issue goes away and the task succeeds. Is there a compatibility problem between django-celery-results==1.0.1 and celery==4.2.0?

using Cache backends requires the app to be in INSTALLED_APPS

I would like to use the CacheBackend but in order to do that I need to have this app registered as INSTALLED_APPS.
this means that migrate will be creating database table for DatabaseBackend which I'm not using.
this is happening because in django_celery_results.backends the module imports DatabaseBackend, and this is imported even if it's unused.

this triggers the import of TaskResult, which triggers an error from new in django.

    from django_celery_results.backends.cache import CacheBackend as DjangoCacheBackend
  File "/usr/local/rotostampa/workflow-local/env/lib/python3.6/site-packages/django_celery_results/backends/__init__.py", line 4, in <module>
    from .database import DatabaseBackend
  File "/usr/local/rotostampa/workflow-local/env/lib/python3.6/site-packages/django_celery_results/backends/database.py", line 7, in <module>
    from ..models import TaskResult
  File "/usr/local/rotostampa/workflow-local/env/lib/python3.6/site-packages/django_celery_results/models.py", line 17, in <module>
    class TaskResult(models.Model):
  File "/usr/local/rotostampa/workflow-local/env/lib/python3.6/site-packages/django/db/models/base.py", line 113, in __new__
    "INSTALLED_APPS." % (module, name)
RuntimeError: Model class django_celery_results.models.TaskResult doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

can you remove the import of DatabaseBackend from there or remove the import of TaskResult?

thanks!

Additional quotes are added when result is string

Checklist

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.

Steps to reproduce

run celery worker with custom class without setting conf.result_serializer

class MyWorker(Worker):
    def run(self):
        # return 100
        return "a test"

Then launch a monitor, observe the result field of events with task-succeeded type

Expected behavior

• when returning 100

{
    "result": 100
}

• when returning "a test"

{
    "result": "a test"
}

Actual behavior

• when returning 100

{
    "result": 100
}

• when returning "a test"

{
    "result": "'a test'"
}

Maybe this is just dumb on my part. But why are extra quotes added when result is a string?

AppRegistryNotReady - apps aren't loaded yet

I receive an AppRegistryNotReady exception when starting up the celery worker using celery worker -A myapp.celery. I am the following versions:

django-celery-results (1.0.1)
celery (4.0.2)

My CELERY_RESULT_BACKEND is 'django-db'.django_celery_results is in INSTALLED_APPS.

I understand why this is happening... I'm not starting up Celery in the context of a manage.py script. But I'm wondering how I'm supposed to start celery if I'm using the DB backend with django-celery-results.

How should I be starting up the celery worker using django-celery-results?

The full traceback:

[2016-12-21 21:56:51,028: CRITICAL/MainProcess] Unrecoverable error: AppRegistryNotReady("Apps aren't loaded yet.",)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 115, in start
    self.on_start()
  File "/usr/local/lib/python2.7/site-packages/celery/apps/worker.py", line 143, in on_start
    self.emit_banner()
  File "/usr/local/lib/python2.7/site-packages/celery/apps/worker.py", line 158, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/usr/local/lib/python2.7/site-packages/celery/apps/worker.py", line 221, in startup_info
    results=self.app.backend.as_uri(),
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python2.7/site-packages/celery/app/base.py", line 1182, in backend
    return self._get_backend()
  File "/usr/local/lib/python2.7/site-packages/celery/app/base.py", line 900, in _get_backend
    self.loader)
  File "/usr/local/lib/python2.7/site-packages/celery/app/backends.py", line 65, in by_url
    return by_name(backend, loader), url
  File "/usr/local/lib/python2.7/site-packages/celery/app/backends.py", line 45, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/site-packages/django_celery_results/backends/__init__.py", line 4, in <module>
    from .database import DatabaseBackend
  File "/usr/local/lib/python2.7/site-packages/django_celery_results/backends/database.py", line 7, in <module>
    from ..models import TaskResult
  File "/usr/local/lib/python2.7/site-packages/django_celery_results/models.py", line 17, in <module>
    class TaskResult(models.Model):
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 94, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 239, in get_containing_app_config
    self.check_apps_ready()
  File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
AppRegistryNotReady: Apps aren't loaded yet.

NotImplementedError: No result backend is configured.

I'm using django, postgres, ubuntu, and (obviously) celery + django-celery-results. I'm following the django guide on the official celery docs that links to this package. In my app's settings.py, CELERY_RESULT_BACKEND is set to 'django-db', and when I check my postgres database, I do see a table for public.django_celery_results_taskresult correctly populating with results, but when I try to access them from a python shell in the following manner:

from tasks import add
result = add.delay(5, 5) 
result.get()

I get the error "NotImplementedError: No result backend is configured."

For the life of me, I can neither figure out why this is happening nor find any answers online.

My celery.py file is:

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'BeerAggregate.settings')

app = Celery('BeerAggregate')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

Finally, when I access the celery shell through

celery -A BeerAggregate -l info

The config message looks like this:

[config]
.> app:         BeerAggregate:0x7f5f0e4159e8
.> transport:   amqp://guest:**@localhost:5672//
.> results:     
.> concurrency: 4 (prefork)
.> task events: OFF (enable -E to monitor tasks in this worker)

The results section is empty

New release?

Hi - now that Celery 4.2 is released, is it possible to do a new release? I've been using a Celery 4.2 pre-release for a while and had to point to a later django-celery-results commit for it to work (4cc5f27); I don't recall why (it wouldn't have been due to that specific commit; that was probably just the latest at the time I did it).

Last release was quite some time ago.

Use multiple Database Backends with celery

Is it possible to use the django-celery-results on a Django installation and the database backend backend on a worker or are they incompatible?
Is it possible to configure them to be compatible?

Or would it be better to have a custom table to share results?

The problem that arises when just using celery and the database backend on the django end is that it will bypass all the nice django features and optimizations. As well as not being noticed by the debug-toolbar.

UUIDField instead of sequence id

This is not currently causing me any real issue, but I was wondering if there is a reason to have an id other than the task_id from celery itself.

You save a sequence a column and index.

Support for revoked tasks

Is this library supposed to be able to support task revocation? I've run into a problem where Celery flags the task as revoked, states that it's discarding it, but then attempts to run it. Additionally, since revoked tasks don't get their arguments passed along from Celery, the task is called without them and causes exceptions to occur. I've traced this back to my use of the django-celery-results library.

When a task is revoked, Celery's mark_as_revoked() method makes a call to store_result() to store the revocation as the result, provided the task's call didn't have ignore_result set to True. This eventually calls into django-celery-results' store_result() which calls into Django to create an instance of (and run!) the task.

Versions:

• Celery: 4.2.1
• Django: 1.11.16
• django-celery-results: master

Broker: RabbitMQ
Result Backend: django-db

Simple reproduction code:
tasks.py:

from __future__ import absolute_import, unicode_literals
from django_celery.celery import app

@app.task
def add(x, y):
    return x + y

@app.task
def add_no_crash(*args, **kwargs):
    print('In add_no_crash()')

    if len(args) == 2:
        x = args[0]
        y = args[1]
        return x + y

    return None

The second task is simply to show that it does actually enter and run the task.

A simple task runner:

import django
import os
import datetime

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_celery.settings")
django.setup()

from celery_test.tasks import add
from celery.result import AsyncResult
from time import sleep

now = datetime.datetime.utcnow()
runat = now + datetime.timedelta(seconds=30)
print('Time is: {}'.format(now))
print('Task scheduled to run at: {}'.format(runat))
result = add.apply_async((1,2), eta=runat)
print('Task ID: {}'.format(result.task_id))
sleep(5)
print('Revoking task')
AsyncResult(result.task_id).revoke()

Results:

[2018-11-11 22:08:34,323: INFO/MainProcess] Received task: celery_test.tasks.add[7a4bf100-ca02-4e32-ab9d-a6552e9e939e]  ETA:[2018-11-11 22:09:34.290097+00:00] 
[2018-11-11 22:08:34,335: INFO/MainProcess] Tasks flagged as revoked: 7a4bf100-ca02-4e32-ab9d-a6552e9e939e
[2018-11-11 22:09:34,290: INFO/MainProcess] Discarding revoked task: celery_test.tasks.add[7a4bf100-ca02-4e32-ab9d-a6552e9e939e]
[2018-11-11 22:09:34,317: ERROR/MainProcess] Error in timer: TypeError("add() missing 2 required positional arguments: 'x' and 'y'",)
Traceback (most recent call last):
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django/db/models/query.py", line 464, in get_or_create
    return self.get(**lookup), False
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
django_celery_results.models.DoesNotExist: TaskResult matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 136, in fire_timers
    entry()
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 477, in apply_eta_task
    self.on_task_request(task)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/worker/worker.py", line 223, in _process_task_sem
    return self._quick_acquire(self._process_task, req)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/kombu/asynchronous/semaphore.py", line 62, in acquire
    callback(*partial_args, **partial_kwargs)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/worker/worker.py", line 228, in _process_task
    req.execute_using_pool(self.pool)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/worker/request.py", line 518, in execute_using_pool
    if (self.expires or task_id in revoked_tasks) and self.revoked():
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/worker/request.py", line 280, in revoked
    'expired' if expired else 'revoked', False, None, expired,
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/worker/request.py", line 263, in _announce_revoked
    self.id, reason, request=self, store_result=self.store_errors,
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/backends/base.py", line 191, in mark_as_revoked
    traceback=None, request=request)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/backends/base.py", line 322, in store_result
    request=request, **kwargs)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django_celery_results/backends/database.py", line 35, in _store_result
    task_kwargs=task_kwargs,
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django_celery_results/managers.py", line 50, in _inner
    return fun(*args, **kwargs)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django_celery_results/managers.py", line 126, in store_result
    obj, created = self.get_or_create(task_id=task_id, defaults=fields)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django/db/models/query.py", line 466, in get_or_create
    return self._create_object_from_params(lookup, params)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django/db/models/query.py", line 499, in _create_object_from_params
    params = {k: v() if callable(v) else v for k, v in params.items()}
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/django/db/models/query.py", line 499, in <dictcomp>
    params = {k: v() if callable(v) else v for k, v in params.items()}
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/app/trace.py", line 642, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/home/noxwizard/.virtualenvs/django-celery/lib/python3.6/site-packages/celery/app/task.py", line 375, in __call__
    return self.run(*args, **kwargs)
TypeError: add() missing 2 required positional arguments: 'x' and 'y'

For the time being, I've written a decorator that I use on each task that will automatically check to see if the task has been revoked and then bail if required. It would be nice to not need it though.

a demo ?

Result Task Admin interface

Accepted Unit Testing patterns fail to store result

Please let me know if I am missing something.

Celery tasks should generally be set to ALWAYS_EAGER for unit testing
http://docs.celeryproject.org/en/latest/userguide/testing.html?highlight=unit%20test

I've been trying this in my unit tests, and the results dont appear in the back end, just the successful task. Any recommendations on getting the task results to store properly when celery is always eager?

How to configure expiration time?

I need to store my tasks only for 3 days.
But then I want them to be expired.

Previously we had CELERY_TASK_RESULT_EXPIRES. Does it still work?

If yes, maybe it would be a good idea to update docs about that?
If no, how we suppose to cleanup tasks?
Thanks!

Relation "django_celery_beat_periodictask" does not exist

Relevant versions:

python:3.5-slim
postgres:9.6-alpine
redis:4.0-alpine
django==1.11.4
celery==4.1.0
django-celery-beat==1.0.1
django-celery-results==1.0.1

Relevant python manage.py migrate output:

...
...
Applying django_celery_beat.0001_initial... OK
Applying django_celery_results.0001_initial... OK
...
...

Relevant settings.py:

CELERY_REDIS_HOSTNAME = env('CELERY_REDIS_HOSTNAME', default='redis_celery')
CELERY_REDIS_PORT = env('CELERY_REDIS_PORT', default=6379)
CELERY_REDIS_DB = env('CELERY_REDIS_DB', default=0)
CELERY_BROKER_URL = env('CELERY_BROKER_URL', default='')

if not CELERY_BROKER_URL:
    CELERY_BROKER_URL = 'redis://{hostname}:{port}/{db}'.format(
        hostname=CELERY_REDIS_HOSTNAME,
        port=CELERY_REDIS_PORT,
        db=CELERY_REDIS_DB
    )

CELERY_BROKER_POOL_LIMIT = 1
CELERY_BROKER_CONNECTION_TIMEOUT = 10

CELERY_ENABLE_UTC = True
CELERY_TIMEZONE = 'UTC'

CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default')
)

CELERY_ALWAYS_EAGER = False
CELERY_ACKS_LATE = True
CELERY_TASK_PUBLISH_RETRY = True
CELERY_DISABLE_RATE_LIMITS = False

CELERY_RESULT_BACKEND = 'django-db'

INSTALLED_APPS of course has django_celery_beat, django_celery_results (that's why it is seen in the migrate output).

project_name/celery.py:

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local')

app = Celery('turnateme')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

Here's the relevant docker-compose up (local development) output:

djangodev_1        | System check identified no issues (0 silenced).
celerybeatdev_1    | Postgres is up - continuing...
celeryworkerdev_1  | Postgres is up - continuing...
celeryworkerdev_1  | /usr/local/lib/python3.5/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is
celeryworkerdev_1  | absolutely not recommended!
celeryworkerdev_1  | 
celeryworkerdev_1  | Please specify a different user using the -u option.
celeryworkerdev_1  | 
celeryworkerdev_1  | User information: uid=0 euid=0 gid=0 egid=0
celeryworkerdev_1  | 
celeryworkerdev_1  |   uid=uid, euid=euid, gid=gid, egid=egid,
celeryworkerdev_1  |  
celeryworkerdev_1  |  -------------- default@4d8350ed217b v4.1.0 (latentcall)
celeryworkerdev_1  | ---- **** ----- 
celeryworkerdev_1  | --- * ***  * -- Linux-4.4.66-boot2docker-x86_64-with-debian-8.9 2017-08-12 20:11:55
celeryworkerdev_1  | -- * - **** --- 
celeryworkerdev_1  | - ** ---------- [config]
celeryworkerdev_1  | - ** ---------- .> app:         turnateme:0x7fdd92f53b70
celeryworkerdev_1  | - ** ---------- .> transport:   redis://redis_celery:6379/0
celeryworkerdev_1  | - ** ---------- .> results:     
celeryworkerdev_1  | - *** --- * --- .> concurrency: 1 (prefork)
celeryworkerdev_1  | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celeryworkerdev_1  | --- ***** ----- 
celeryworkerdev_1  |  -------------- [queues]
celeryworkerdev_1  |                 .> default          exchange=default(direct) key=default
celeryworkerdev_1  |                 
celeryworkerdev_1  | 
celeryworkerdev_1  | [tasks]
celeryworkerdev_1  |   . billing.tasks.charge_period
celeryworkerdev_1  |   . billing.tasks.finish_period
celeryworkerdev_1  |   . billing.tasks.process_webhook_event
celeryworkerdev_1  |   . turnateme.celery.debug_task
celeryworkerdev_1  | 
celeryworkerdev_1  | [2017-08-12 20:11:55,228: INFO/MainProcess] Connected to redis://redis_celery:6379/0
celerybeatdev_1    | celery beat v4.1.0 (latentcall) is starting.
celerybeatdev_1    | __    -    ... __   -        _
celerybeatdev_1    | LocalTime -> 2017-08-12 20:11:55
celerybeatdev_1    | Configuration ->
celerybeatdev_1    |     . broker -> redis://redis_celery:6379/0
celerybeatdev_1    |     . loader -> celery.loaders.app.AppLoader
celerybeatdev_1    |     . scheduler -> django_celery_beat.schedulers.DatabaseScheduler
celerybeatdev_1    | 
celerybeatdev_1    |     . logfile -> [stderr]@%INFO
celerybeatdev_1    |     . maxinterval -> 5.00 seconds (5s)
celerybeatdev_1    | [2017-08-12 20:11:55,520: INFO/MainProcess] beat: Starting...
celerybeatdev_1    | [2017-08-12 20:11:55,521: INFO/MainProcess] Writing entries...
postgres_1         | ERROR:  relation "django_celery_beat_periodictask" does not exist at character 748
postgres_1         | STATEMENT:  SELECT "django_celery_beat_periodictask"."id", "django_celery_beat_periodictask"."name", "django_celery_beat_periodictask"."task", "django_celery_beat_periodictask"."interval_id", "django_celery_beat_periodictask"."crontab_id", "django_celery_beat_periodictask"."args", "django_celery_beat_periodictask"."kwargs", "django_celery_beat_periodictask"."queue", "django_celery_beat_periodictask"."exchange", "django_celery_beat_periodictask"."routing_key", "django_celery_beat_periodictask"."expires", "django_celery_beat_periodictask"."enabled", "django_celery_beat_periodictask"."last_run_at", "django_celery_beat_periodictask"."total_run_count", "django_celery_beat_periodictask"."date_changed", "django_celery_beat_periodictask"."description" FROM "django_celery_beat_periodictask" WHERE "django_celery_beat_periodictask"."enabled" = true
celerybeatdev_1    | [2017-08-12 20:11:55,538: CRITICAL/MainProcess] beat raised exception <class 'django.db.utils.ProgrammingError'>: ProgrammingError('relation "django_celery_beat_periodictask" does not exist\nLINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...\n                                                             ^\n',)
celerybeatdev_1    | Traceback (most recent call last):
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/kombu/utils/objects.py", line 42, in __get__
celerybeatdev_1    |     return obj.__dict__[self.__name__]
celerybeatdev_1    | KeyError: 'scheduler'
celerybeatdev_1    | 
celerybeatdev_1    | During handling of the above exception, another exception occurred:
celerybeatdev_1    | 
celerybeatdev_1    | Traceback (most recent call last):
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
celerybeatdev_1    |     return self.cursor.execute(sql, params)
celerybeatdev_1    | psycopg2.ProgrammingError: relation "django_celery_beat_periodictask" does not exist
celerybeatdev_1    | LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
celerybeatdev_1    |                                                              ^
celerybeatdev_1    | 
celerybeatdev_1    | 
celerybeatdev_1    | The above exception was the direct cause of the following exception:
celerybeatdev_1    | 
celerybeatdev_1    | Traceback (most recent call last):
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/celery/apps/beat.py", line 107, in start_scheduler
celerybeatdev_1    |     service.start()
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 549, in start
celerybeatdev_1    |     humanize_seconds(self.scheduler.max_interval))
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/kombu/utils/objects.py", line 44, in __get__
celerybeatdev_1    |     value = obj.__dict__[self.__name__] = self.__get(obj)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 593, in scheduler
celerybeatdev_1    |     return self.get_scheduler()
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 588, in get_scheduler
celerybeatdev_1    |     lazy=lazy,
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 179, in __init__
celerybeatdev_1    |     Scheduler.__init__(self, *args, **kwargs)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 206, in __init__
celerybeatdev_1    |     self.setup_schedule()
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 187, in setup_schedule
celerybeatdev_1    |     self.install_default_entries(self.schedule)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 280, in schedule
celerybeatdev_1    |     self._schedule = self.all_as_schedule()
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 193, in all_as_schedule
celerybeatdev_1    |     for model in self.Model.objects.enabled():
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 250, in __iter__
celerybeatdev_1    |     self._fetch_all()
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 1118, in _fetch_all
celerybeatdev_1    |     self._result_cache = list(self._iterable_class(self))
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 53, in __iter__
celerybeatdev_1    |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
celerybeatdev_1    |     raise original_exception
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 884, in execute_sql
celerybeatdev_1    |     cursor.execute(sql, params)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 80, in execute
celerybeatdev_1    |     return super(CursorDebugWrapper, self).execute(sql, params)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
celerybeatdev_1    |     return self.cursor.execute(sql, params)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
celerybeatdev_1    |     six.reraise(dj_exc_type, dj_exc_value, traceback)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
celerybeatdev_1    |     raise value.with_traceback(tb)
celerybeatdev_1    |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
celerybeatdev_1    |     return self.cursor.execute(sql, params)
celerybeatdev_1    | django.db.utils.ProgrammingError: relation "django_celery_beat_periodictask" does not exist
celerybeatdev_1    | LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
celerybeatdev_1    |                                                              ^
celerybeatdev_1    | 
celerybeatdev_1    | [2017-08-12 20:11:55,540: WARNING/MainProcess] Traceback (most recent call last):
celerybeatdev_1    | [2017-08-12 20:11:55,540: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/kombu/utils/objects.py", line 42, in __get__
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] return obj.__dict__[self.__name__]
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] KeyError
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] :
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] 'scheduler'
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] During handling of the above exception, another exception occurred:
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] Traceback (most recent call last):
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
celerybeatdev_1    | [2017-08-12 20:11:55,541: WARNING/MainProcess] return self.cursor.execute(sql, params)
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] psycopg2
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] .
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] ProgrammingError
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] :
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] relation "django_celery_beat_periodictask" does not exist
celerybeatdev_1    | LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
celerybeatdev_1    |                                                              ^
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] The above exception was the direct cause of the following exception:
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] Traceback (most recent call last):
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] File "/usr/local/bin/celery", line 11, in <module>
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] sys.exit(main())
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/__main__.py", line 14, in main
celerybeatdev_1    | [2017-08-12 20:11:55,542: WARNING/MainProcess] _main()
celerybeatdev_1    | [2017-08-12 20:11:55,543: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 326, in main
celerybeatdev_1    | [2017-08-12 20:11:55,543: WARNING/MainProcess] cmd.execute_from_commandline(argv)
celerybeatdev_1    | [2017-08-12 20:11:55,543: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
celerybeatdev_1    | [2017-08-12 20:11:55,543: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
celerybeatdev_1    | [2017-08-12 20:11:55,544: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
celerybeatdev_1    | [2017-08-12 20:11:55,544: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
celerybeatdev_1    | [2017-08-12 20:11:55,544: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 480, in handle_argv
celerybeatdev_1    | [2017-08-12 20:11:55,544: WARNING/MainProcess] return self.execute(command, argv)
celerybeatdev_1    | [2017-08-12 20:11:55,544: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/celery.py", line 412, in execute
celerybeatdev_1    | [2017-08-12 20:11:55,545: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
celerybeatdev_1    | [2017-08-12 20:11:55,545: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/base.py", line 285, in run_from_argv
celerybeatdev_1    | [2017-08-12 20:11:55,545: WARNING/MainProcess] sys.argv if argv is None else argv, command)
celerybeatdev_1    | [2017-08-12 20:11:55,545: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/base.py", line 368, in handle_argv
celerybeatdev_1    | [2017-08-12 20:11:55,545: WARNING/MainProcess] return self(*args, **options)
celerybeatdev_1    | [2017-08-12 20:11:55,545: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/base.py", line 244, in __call__
celerybeatdev_1    | [2017-08-12 20:11:55,546: WARNING/MainProcess] ret = self.run(*args, **kwargs)
celerybeatdev_1    | [2017-08-12 20:11:55,546: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/bin/beat.py", line 107, in run
celerybeatdev_1    | [2017-08-12 20:11:55,546: WARNING/MainProcess] return beat().run()
celerybeatdev_1    | [2017-08-12 20:11:55,546: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/apps/beat.py", line 79, in run
celerybeatdev_1    | [2017-08-12 20:11:55,546: WARNING/MainProcess] self.start_scheduler()
celerybeatdev_1    | [2017-08-12 20:11:55,547: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/apps/beat.py", line 107, in start_scheduler
celerybeatdev_1    | [2017-08-12 20:11:55,547: WARNING/MainProcess] service.start()
celerybeatdev_1    | [2017-08-12 20:11:55,547: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 549, in start
celerybeatdev_1    | [2017-08-12 20:11:55,547: WARNING/MainProcess] humanize_seconds(self.scheduler.max_interval))
celerybeatdev_1    | [2017-08-12 20:11:55,547: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/kombu/utils/objects.py", line 44, in __get__
celerybeatdev_1    | [2017-08-12 20:11:55,547: WARNING/MainProcess] value = obj.__dict__[self.__name__] = self.__get(obj)
celerybeatdev_1    | [2017-08-12 20:11:55,547: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 593, in scheduler
celerybeatdev_1    | [2017-08-12 20:11:55,548: WARNING/MainProcess] return self.get_scheduler()
celerybeatdev_1    | [2017-08-12 20:11:55,548: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 588, in get_scheduler
celerybeatdev_1    | [2017-08-12 20:11:55,548: WARNING/MainProcess] lazy=lazy,
celerybeatdev_1    | [2017-08-12 20:11:55,548: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 179, in __init__
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] Scheduler.__init__(self, *args, **kwargs)
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/celery/beat.py", line 206, in __init__
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] self.setup_schedule()
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 187, in setup_schedule
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] self.install_default_entries(self.schedule)
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 280, in schedule
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] self._schedule = self.all_as_schedule()
celerybeatdev_1    | [2017-08-12 20:11:55,549: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django_celery_beat/schedulers.py", line 193, in all_as_schedule
celerybeatdev_1    | [2017-08-12 20:11:55,550: WARNING/MainProcess] for model in self.Model.objects.enabled():
celerybeatdev_1    | [2017-08-12 20:11:55,550: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 250, in __iter__
celerybeatdev_1    | [2017-08-12 20:11:55,550: WARNING/MainProcess] self._fetch_all()
celerybeatdev_1    | [2017-08-12 20:11:55,550: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 1118, in _fetch_all
celerybeatdev_1    | [2017-08-12 20:11:55,551: WARNING/MainProcess] self._result_cache = list(self._iterable_class(self))
celerybeatdev_1    | [2017-08-12 20:11:55,551: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 53, in __iter__
celerybeatdev_1    | [2017-08-12 20:11:55,551: WARNING/MainProcess] results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
celerybeatdev_1    | [2017-08-12 20:11:55,551: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
celerybeatdev_1    | [2017-08-12 20:11:55,560: WARNING/MainProcess] raise original_exception
celerybeatdev_1    | [2017-08-12 20:11:55,560: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 884, in execute_sql
celerybeatdev_1    | [2017-08-12 20:11:55,561: WARNING/MainProcess] cursor.execute(sql, params)
celerybeatdev_1    | [2017-08-12 20:11:55,561: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 80, in execute
celerybeatdev_1    | [2017-08-12 20:11:55,561: WARNING/MainProcess] return super(CursorDebugWrapper, self).execute(sql, params)
celerybeatdev_1    | [2017-08-12 20:11:55,561: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
celerybeatdev_1    | [2017-08-12 20:11:55,561: WARNING/MainProcess] return self.cursor.execute(sql, params)
celerybeatdev_1    | [2017-08-12 20:11:55,562: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
celerybeatdev_1    | [2017-08-12 20:11:55,562: WARNING/MainProcess] six.reraise(dj_exc_type, dj_exc_value, traceback)
celerybeatdev_1    | [2017-08-12 20:11:55,562: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
celerybeatdev_1    | [2017-08-12 20:11:55,562: WARNING/MainProcess] raise value.with_traceback(tb)
celerybeatdev_1    | [2017-08-12 20:11:55,564: WARNING/MainProcess] File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
celerybeatdev_1    | [2017-08-12 20:11:55,564: WARNING/MainProcess] return self.cursor.execute(sql, params)
celerybeatdev_1    | [2017-08-12 20:11:55,564: WARNING/MainProcess] django.db.utils
celerybeatdev_1    | [2017-08-12 20:11:55,564: WARNING/MainProcess] .
celerybeatdev_1    | [2017-08-12 20:11:55,564: WARNING/MainProcess] ProgrammingError
celerybeatdev_1    | [2017-08-12 20:11:55,564: WARNING/MainProcess] :
celerybeatdev_1    | [2017-08-12 20:11:55,564: WARNING/MainProcess] relation "django_celery_beat_periodictask" does not exist
celerybeatdev_1    | LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
celerybeatdev_1    |                                                              ^
celerybeatdev_1    | [2017-08-12 20:11:55,564: INFO/MainProcess] Writing entries...
turnateme_celerybeatdev_1 exited with code 1
celeryworkerdev_1  | [2017-08-12 20:11:55,240: INFO/MainProcess] mingle: searching for neighbors
celeryworkerdev_1  | [2017-08-12 20:11:56,273: INFO/MainProcess] mingle: all alone
celeryworkerdev_1  | [2017-08-12 20:11:56,284: WARNING/MainProcess] /usr/local/lib/python3.5/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!

If anything else is needed to further debug this, please let me know. Thank you in advance.

ProgrammingError: relation "celery_results_taskresult" does not exist LINE 1: SELECT COUNT(*) AS "__count" FROM "celery_results_taskresult...

When I tried out the master version (1.0.2), I receive the following error when visiting /admin/celery_results/taskresult.

ProgrammingError: relation "celery_results_taskresult" does not exist
LINE 1: SELECT COUNT(*) AS "__count" FROM "celery_results_taskresult...

Chords not supported when using 'django-db' backend

I'm trying a simple chord example while using the 'django-db' result backend, and get back this error:

Traceback (most recent call last):
  File "/Users/pavel/dev/test_proj/src/batch/tests/test_batch.py", line 51, in test_stringing
    queue_batch()
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/app/task.py", line 380, in __call__
    return self.run(*args, **kwargs)
  File "/Users/pavel/dev/test_proj/src/batch/tasks.py", line 218, in queue_batch
    schedule_tasks(tasks, object)
  File "/Users/pavel/dev/test_proj/src/batch/tasks.py", line 196, in schedule_tasks
    chord(all_tasks_group)(set_object_state.s(object_id=object.id)).delay()
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/canvas.py", line 1196, in __call__
    return self.apply_async((), {'body': body} if body else {}, **options)
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/canvas.py", line 1237, in apply_async
    body=body, task_id=task_id, **options)
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/canvas.py", line 1251, in apply
    args=(tasks.apply(args, kwargs).get(propagate=propagate),),
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/result.py", line 635, in get
    on_message=on_message,
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/result.py", line 746, in join_native
    on_message, on_interval):
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/result.py", line 724, in iter_native
    on_message=on_message, on_interval=on_interval,
  File "/Users/pavel.virtualenvs/test_proj/lib/python3.6/site-packages/celery/backends/base.py", line 448, in iter_native
    return self.get_many(
AttributeError: 'DatabaseBackend' object has no attribute 'get_many'

Does anyone know if this is possible, and how can I work around this?

missing migration

with a fresh install (django version 2.1.2) I'm getting a migration not committed to source:

Migrations for 'celery_results':
  /path/to/virtualenv/lib/python3.6/site-packages/django_celery_results/migrations/0003_auto_20181004_1724.py
    - Change Meta options on taskresult

Quick Question: Using the cache backend, when does the result expire?

How does expiry work when using the cache result backend?

traceback is null

I am having several tasks which are being reported with status FAILURE however the traceback field in the database result is null. Shouldn't the traceback contain a stack trace?

Another (potentially related) issue is that the result field is containing some strange unreadable data. Here are some sample rows.

Any help into diagnosing the issue would be appreciated.

devtaskresult=> select * from celery_taskmeta where status = 'FAILURE' and date_done > '2018-10-05 03:00' and date_done < '2018-10-05 04:00' order by id asc;        
-[ RECORD 1 ]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id        | 6080263
task_id   | e1425dc2-00d6-42e6-8069-21cbe034fcac
status    | FAILURE
result    | \x80027d710128580b0000006578635f6d6573736167657102581900000054696d654c696d69744578636565646564283330302e302c2958080000006578635f747970657103551154696d654c696d697445786365656465647104752e
date_done | 2018-10-05 03:05:45.301703
traceback |
-[ RECORD 2 ]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id        | 6080264
task_id   | 0fe13afe-2947-4765-8309-74ddd1132688
status    | FAILURE
result    | \x80027d710128580b0000006578635f6d6573736167657102581900000054696d654c696d69744578636565646564283330302e302c2958080000006578635f747970657103551154696d654c696d697445786365656465647104752e
date_done | 2018-10-05 03:10:47.547563
traceback |

devtaskresult=> select * from celery_taskmeta  where task_id = 'a9d85f18-6db8-4261-b88d-73e683e4c1f4';
-[ RECORD 1 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id        | 6115068
task_id   | a9d85f18-6db8-4261-b88d-73e683e4c1f4
status    | FAILURE
result    | \x80027d710128580b0000006578635f6d6573736167657102474072c0000000000085580a0000006578635f6d6f64756c657103551362696c6c696172642e657863657074696f6e73710458080000006578635f747970657105551154696d654c696d697445786365656465647106752e
date_done | 2018-10-05 13:08:57.323079
traceback |

DecodeError: "invalid load key" with CELERY_RESULT_SERIALIZER = 'pickle'

Django 2.0.3, SQLite database, django-celery-results 1.0.1

Here is what it's trying to unpickle:

_trusted_content | frozenset({'application/data', 'application/text'})
-- | --
accept | {'application/json', 'application/x-python-serialize'}
content_encoding | 'binary'
content_type | 'application/x-python-serialize'
data | (b'n\x00\t\xf5\xc4\x01` \x00\x00\x01\x8d\xa1\xa5\xb1\x91\xc9\x95\xb9\xc4'  b'\x05u\xc4\t\xcc\xb8')
decode | <function unpickle at 0x7f9b91935d08>
force | False
self | <kombu.serialization.SerializerRegistry object at 0x7f9b91938518>

Sample traceback:

Environment:

Request Method: GET
Request URL: http://localhost:8000/monitor-task/9d49ffda-9f80-4862-a8b6-ed23b8f50224

Django Version: 2.0.3
Python Version: 3.6.5
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'crispy_forms',
 'jsonfield',
 'bootstrap3_datetime',
 'django_select2',
 'django_celery_results',
 'accounts',
 'course')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'course.auth.ImpersonateMiddleware',
 'course.utils.FacilityFindingMiddleware',
 'course.exam.ExamFacilityMiddleware',
 'course.exam.ExamLockdownMiddleware',
 'relate.utils.MaintenanceMiddleware')



Traceback:

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in _reraise_errors
  50.         yield

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in loads
  263.                     return decode(data)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in unpickle
  337.         return pickle_loads(str_to_bytes(s))

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in pickle_loads
  59.     return load(BytesIO(s))

During handling of the above exception (invalid load key, 'n'.), another exception occurred:

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  35.             response = get_response(request)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  128.                 response = self.process_exception_by_middleware(e, request)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  21.                 return view_func(request, *args, **kwargs)

File "/home/andreas/work/django/relate/course/views.py" in monitor_task
  1317.     if async_res.state == "PROGRESS":

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/celery/result.py" in state
  436.         return self._get_task_meta()['status']

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/celery/result.py" in _get_task_meta
  375.             return self._maybe_set_cache(self.backend.get_task_meta(self.id))

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/celery/backends/base.py" in get_task_meta
  352.         meta = self._get_task_meta_for(task_id)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/django_celery_results/backends/database.py" in _get_task_meta_for
  37.         meta = self.decode_content(obj, res.pop('meta', None)) or {}

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/django_celery_results/backends/database.py" in decode_content
  52.             return self.decode(content)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/celery/backends/base.py" in decode
  278.                      accept=self.accept)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in loads
  263.                     return decode(data)

File "/usr/lib/python3.6/contextlib.py" in __exit__
  99.                 self.gen.throw(type, value, traceback)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in _reraise_errors
  54.         reraise(wrapper, wrapper(exc), sys.exc_info()[2])

File "/usr/lib/python3/dist-packages/vine/five.py" in reraise
  178.             raise value.with_traceback(tb)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in _reraise_errors
  50.         yield

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in loads
  263.                     return decode(data)

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in unpickle
  337.         return pickle_loads(str_to_bytes(s))

File "/home/andreas/src/env-3.6/lib/python3.6/site-packages/kombu/serialization.py" in pickle_loads
  59.     return load(BytesIO(s))

Exception Type: DecodeError at /monitor-task/9d49ffda-9f80-4862-a8b6-ed23b8f50224
Exception Value: invalid load key, 'n'.

relation "task_id_sequence" does not exist

Celery fails to save data into database backend

Stack
celery==4.1.0
django-celery==3.2.2
django-celery-results==1.0.1

Run migrations
python3 manage.py migrate djcelery
python3 manage.py migrate django_celery_results

This is schema for celery_taskmeta table ("task_id_sequence" does not exist)

id 	    integer               NOT NULL nextval('celery_taskmeta_id_seq'::regclass) 	[pk]
task_id     character varying(255)    NOT NULL [uniq] 
status 	    character varying(50)     NOT NULL
result 	    text
date_done 	timestamp with time zone  NOT NULL
traceback 	text
hidden 	        boolean                   NOT NULL
meta 	      text

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/billiard/pool.py", line 1747, in safe_apply_callback
    fun(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/celery/worker/request.py", line 366, in on_failure
    self.id, exc, request=self, store_result=self.store_errors,
  File "/usr/local/lib/python3.5/dist-packages/celery/backends/base.py", line 163, in mark_as_failure
    traceback=traceback, request=request)
  File "/usr/local/lib/python3.5/dist-packages/celery/backends/base.py", line 309, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/celery/backends/database/__init__.py", line 112, in _store_result
    session.add(task)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2237, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2363, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 2327, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
    uow
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py", line 866, in _emit_insert_statements
    execute(statement, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "task_id_sequence" does not exist
LINE 1: ...us, result, date_done, traceback) VALUES (nextval('task_id_s...
                                                             ^
 [SQL: "INSERT INTO celery_taskmeta (id, task_id, status, result, date_done, traceback) VALUES (nextval('task_id_sequence'), %(task_id)s, %(status)s, %(result)s, %(date_done)s, %(traceback)s) RETURNING celery_taskmeta.id"] [parameters: {'date_done': datetime.datetime(2017, 12, 21, 13, 0, 8, 211704), 'result': None, 'traceback': None, 'status': 'PENDING', 'task_id': 'a06b980a-36f8-4fe0-a8c6-d920341a217a'}]

How to fix this compatibility problem ?

How to use it?

When i start celery worker i get error

appregistrynotready: apps aren't loaded yet.

if i change, CELERY_RESULT_BACKEND = 'django-db' to CELERY_RESULT_BACKEND = 'redis://localhost:6379' , celery started, but i don't have task results in Django Admin.

What i do wrong?

App versions

Django (1.10.5)
django-celery-beat (1.0.1)
django-celery-results (1.0.1)
celery (4.0.2)

project/celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery("project_celery")

# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

project/init.py

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
from .tasks import *


__all__ = ['celery_app']

project/settings.py

INSTALLED_APPS = [
    . . . 
    'django_celery_results',
    'django_celery_beat'
]

# CELERY SETTINGS
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'django-db'
CELERY_TIMEZONE = TIME_ZONE
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_TRACK_STARTED = True

What's the use of 'hidden' field?

Worker status in the admin

@ask In django-celery we could see the active workers in the admin, with this new package is there a way for that as well?

Incorrect documentation on seting up `CELERY_CACHE_BACKEND`

The First steps with Django tutorial of the celery documentation describes what variables to set in the settings.py file of a Django project.
In the section "django-celery-results - Using the Django ORM/Cache as a result backend" it indicates that the following settings have to be set:

CELERY_RESULT_BACKEND = 'django-db'
CELERY_RESULT_BACKEND = 'django-cache'

This is obviously incorrect but nonetheless cost me significant time debugging before I spotted the mistake.

The correct settings should be :

CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'

This bug is present in v4.2.1 and latest.

on_message support

The result backend 'django-db' and 'django-cache' doesn't support "on_message" ?!?

I get this error:

Backend does not support on_message callback

EDIT: Hm, ok now i see that all backends in celery except "redit" and "RPC" dosn't support "on_message" :(

Task fail with ImportError: No module named 'django-db'

Current configuration

Celery report:

software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.5.2
billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:django-db

CELERYD_HIJACK_ROOT_LOGGER: False
CELERY_RESULT_BACKEND: 'django-db'
CELERY_RESULT_PERSISTENT: True
CELERY_IGNORE_RESULT: True
CELERY_TASK_RESULT_EXPIRES: datetime.timedelta(7)
CELERY_QUEUE_HA_POLICY: 'all'
CELERY_BROKER_URL: 'amqp://project:********@*******:5672/project'
CELERY_WORKER_DIRECT: False
CELERY_TIMEZONE: 'Europe/Berlin'
CELERY_SEND_EVENTS: False
CELERY_TRACK_STARTED: True

Content of celery.py:

import os
from celery import Celery
from celery.schedules import crontab

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

tasks.py:

@celery_app.task(autoretry_for=(Exception,), retry_kwargs={'max_retries': 5})
def revoke_x(name, id2, id2):
    return

Steps to reproduce

I'm not able to find out how to reproduce it

Expected behavior

Task running

Actual behavior

It's a weird behavior, as it's not failing always, I have many others tasks running with no problems, this problem happen sometimes but only with this task, and I'm not able to understand what's the source of the issue.
Error happening before executing the tasks (When trying to send it to the queue)
The call: revoke_x.apply_async(['param1', 1, 1])

traceback:
ImportError: No module named 'django-db'
KeyError 'backend' kombu/utils/objects.py in get at line 42

    def __get__(self, obj, type=None):
    if obj is None:
    return self
    try:
    return obj.__dict__[self.__name__]
    except KeyError:
    value = obj.__dict__[self.__name__] = self.__get(obj)
    return value
    def __set__(self, obj, value):

ImportError No module named 'django-db'

celery/app/backends.py in by_url at line 65
scheme, _, _ = url.partition('://')
if '+' in scheme:
backend, url = url.split('+', 1)
else:
backend = scheme
return by_name(backend, loader), url

Values in context (From Sentry)
backend: 'django-db'
loader: <celery.loaders.app.AppLoader object at 0x7f13258a2f28>
url: None

Migrations not being Found

For some reason this app is not correctly showing up in the list of migrations for me.

Operations to perform:
  Apply all migrations: django_celery_results
Running migrations:
  No migrations to apply.

Its in INSTALLED_APPS, and even happens when I copy the folder into my project structure to ensure that it is discovered with the rest of my project apps.

how to get task's result from database realtime

I'm using celery 4.0.2, django 1.10, mysql and I add a task like below:

@app.task
def add():
    print 'start add'
    time.sleep(30)
    print 'end add'
    return 1

but in the tabledjango_celery_results_taskresult, I can only get the record after the task excuted (means after 30s). I don't know why, can any one help me

update
final, i find it's my config's problem, after i add CELERY_TASK_TRACK_STARTED=True, it works

Migrations

Hi. I was just wondering if you plan on creating any migrations for this project.

Thanks Stuart

Tasks scheduled for later are not shown in the admin panel.

I created a task for later like this,
task_name.apply_async(args=['a', 'b'], countdown=500)

This should list this newly created task in the django admin panel with the status as PENDING, but it doesn't.
Normal tasks are getting listed as expected.

Is this the intended behaviour?
If yes, Is there any other library which can list the tasks and show their updated status?

django-celery-results doesn't update state correctly with task.update_state()

When using sample code in celery doc:

@app.task(bind=True)
def upload_files(self, filenames):
    for i, file in enumerate(filenames):
        if not self.request.called_directly:
            self.update_state(state='PROGRESS',
                meta={'current': i, 'total': len(filenames)})

The task's state doesn't change, and the meta data is actually written to result field.

This only happens when using CELERY_RESULT_BACKEND = 'django-db'. When using SQLAlchemy and set the backend directly to database, the progress field is updated to PROGRESS correctly.

I'm currently using:

celery (4.1.0)
Django (1.11.4)
django-celery-beat (1.0.1)

My broker is Rabbitmq, and database is PostgreSQL

Latest release number is behind current master

Hi all, some substantial changes have been made in the last 15 months, including the database migration to 0002 in the recent merge #34 .

Is there any chance of a new release (to 1.0.2??) to correspond with this, please? It'd help with tracking migrations and dependencies when pip installing a version, rather than setting a specific commit SHA :)

Thanks!

Migrate from django-celery and keep older results

Hello.
I am trying to migrate from django-celery to django-celery-results.
Django-celery-results is working just fine. But the older results kept in the database in the table celery_taskmeta weren't migrated automatically to django-celery-result table.
Should I do it manually or there is something that I am missing?

Thanks a lot for the help.
Mauro.

Docs link in README 404s

The link to http://django-celery-results.readthedocs.org 404s.

How to get the task_id generated by crontab task

When I run task, I can get the result of task through django_celery_result, but task can not find the one-to-one correspondence with the result.

Wrong application name in 0002 migration

#34 introduced new migration 0002_add_task_name_args_kwargs with following dependencies:

    dependencies = [
        ('celery_results', '0001_initial'),
    ]

but application name is django_celery_results, not celery_results.

That causes following issue, after installing package:

django.db.migrations.exceptions.NodeNotFoundError: Migration django_celery_results.0002_add_task_name_args_kwargs dependencies reference nonexistent parent node ('celery_results', '0001_initial')

Jobs that are enqueued and running

How would I see jobs that are either enqueued or currently running?

Extending TaskResult model

I'm trying to find a way to extend the TaskResult model and add another field to it. I want to add request.user object to the job model in database so that I can query it later in my app.

Is subclassing a correct approach to do this? I'm asking this because subclassing creates another table in the database and then I need to manually sync the status of new model with what django-celery-results create.

Tasks are not getting added to database

Hi,
I have followed this link: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html#django-celery-results-using-the-django-orm-cache-as-a-result-backend
to set up django-celery-results for my project. Migrations are running fine and i can see table django_celery_results_taskresult got created.
The problem is when i try to execute a task, it runs fine but the table i mentioned above didn't get populated.
I have put CELERY_RESULT_BACKEND = 'django-db' and have placed django_celery_results in my INSTALLED_APPS. There is no error in logs either. Can anyone give me hint what i am doing wrong? Or guide me how can i debug this issue?

Celery app fails when importing Django user model

as in snapshot and highlighted text in red I have this line

from gallery.colorfest import render_full

which inside goes to import Django user model and it fails claiming apps not installed. If I un comment this line it works, so with possibly a good amount of code reshuffling I might get it to work but I believe figuring this out will end up making a more streamlined implementation.

AppRegistryNotReady when trying to use CELERY_RESULT_BACKEND = 'django-db'

Hi!

I'm experiencing an issue very similar to what was described in #11, but I can't find a misconfiguration. I have been following http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html so far, and I have:

celery.py

import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')

from celery import Celery

app = Celery('myapp')
app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

And then in settings I have:

INSTALLED_APPS = [
    ....
    'django_celery_results',
]

...

CELERY_RESULT_BACKEND = 'django-db'

I can run a worker successfully, and I can run a task from django shell. What I cannot do is, for example, to inspect the result of the task:

$ celery -A myapp result 3e5d00a5-f20e-4d1b-8996-eaade4b1b051
Traceback (most recent call last):
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/immerrr/.conda/envs/myapp/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 480, in handle_argv
    return self.execute(command, argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 412, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 285, in run_from_argv
    sys.argv if argv is None else argv, command)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 368, in handle_argv
    return self(*args, **options)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 244, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/result.py", line 36, in run
    task_result = result_cls(task_id)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/result.py", line 93, in __init__
    self.backend = backend or self.app.backend
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/base.py", line 1182, in backend
    return self._get_backend()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/base.py", line 900, in _get_backend
    self.loader)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/backends.py", line 65, in by_url
    return by_name(backend, loader), url
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/backends.py", line 45, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django_celery_results/backends/__init__.py", line 4, in <module>
    from .database import DatabaseBackend
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django_celery_results/backends/database.py", line 7, in <module>
    from ..models import TaskResult
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django_celery_results/models.py", line 17, in <module>
    class TaskResult(models.Model):
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django/db/models/base.py", line 110, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django/apps/registry.py", line 247, in get_containing_app_config
    self.check_apps_ready()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django/apps/registry.py", line 125, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

It seems like for some reason the fixup doesn't fire before the backend is accessed.

If it is of any help, accessing the backend happens from here:

  File "/home/immerrr/.conda/envs/myapp/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 480, in handle_argv
    return self.execute(command, argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 412, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 285, in run_from_argv
    sys.argv if argv is None else argv, command)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 368, in handle_argv
    return self(*args, **options)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 244, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/result.py", line 36, in run
    task_result = result_cls(task_id)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/result.py", line 93, in __init__
    self.backend = backend or self.app.backend

Here's my pip freeze:

amqp==2.1.4
anyjson==0.3.3
billiard==3.5.0.2
celery==4.0.2
Cerberus==1.1
click==6.7
coverage==4.3.4
decorator==4.0.11
Django==1.11.1
django-celery-results==1.0.1
djangorestframework==3.6.2
first==2.0.1
httmock==1.2.6
ipython==6.0.0
ipython-genutils==0.2.0
kombu==4.0.2
pexpect==4.2.1
pickleshare==0.7.4
pip-tools==1.9.0
prompt-toolkit==1.0.14
ptyprocess==0.5.1
pudb==2017.1.2
py==1.4.33
Pygments==2.2.0
pytest==3.0.7
pytest-cov==2.4.0
pytest-django==3.1.2
pytz==2017.2
requests==2.13.0
simplegeneric==0.8.1
six==1.10.0
traitlets==4.3.2
urwid==1.3.1
vine==1.1.3
wcwidth==0.1.7

Store task name ?

Hello.

Would there be a way to store the task name with the result ? If not, how to know what is the task associated with a result ?

Thanks.

Job Names

How do I see the name on the job?

Writing results temporarly to memory

Would you consider a new feature (if there is not a similar one already that i couldn't find) which is writing the celery results in a temporary cache backend (like Redis for example), and having a callable Django management command to retrieve multiple results and dump them in bulk as well into a Database (Postgres for e.g)??

This way we could have the "historical" data without massacring our database even if it was not up to date.

In my case i only check these results in Django admin once in a while and before i do, i see myself running the command (or assigning it to celery beat) to get the updates into the database and into the admin.

Not compatible with celery < 4.0

I had installed this with Celery 3.1.23 and got this error:

ImportError: cannot import name python_2_unicode_compatible

but I couldn't find any warning on that in the docs or elsewhere. Even better would be to fix the dependancies in setup.py.