jazzband / django-celery-monitor Goto Github PK

@jezdez i am running into this error:

celerycam_1   | -> evcam: Taking snapshots with django_celery_monitor.camera.Camera (every 2.0 secs.)
celerycam_1   | Traceback (most recent call last):
celerycam_1   |   File "/usr/local/bin/celery", line 11, in <module>
celerycam_1   |     sys.exit(main())
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/__main__.py", line 14, in main
celerycam_1   |     _main()
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 326, in main
celerycam_1   |     cmd.execute_from_commandline(argv)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 488, in execute_from_commandline
celerycam_1   |     super(CeleryCommand, self).execute_from_commandline(argv)))
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 281, in execute_from_commandline
celerycam_1   |     return self.handle_argv(self.prog_name, argv[1:])
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 480, in handle_argv
celerycam_1   |     return self.execute(command, argv)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 412, in execute
celerycam_1   |     ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 285, in run_from_argv
celerycam_1   |     sys.argv if argv is None else argv, command)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 368, in handle_argv
celerycam_1   |     return self(*args, **options)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 244, in __call__
celerycam_1   |     ret = self.run(*args, **kwargs)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/events.py", line 118, in run
celerycam_1   |     detach=detach)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/bin/events.py", line 144, in run_evcam
celerycam_1   |     return cam()
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/events/snapshot.py", line 101, in evcam
celerycam_1   |     maxrate=maxrate, timer=timer)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/celery/utils/imports.py", line 53, in instantiate
celerycam_1   |     return symbol_by_name(name)(*args, **kwargs)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/kombu/utils/imports.py", line 56, in symbol_by_name
celerycam_1   |     module = imp(module_name, package=package, **kwargs)
celerycam_1   |   File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
celerycam_1   |     return _bootstrap._gcd_import(name[level:], package, level)
celerycam_1   |   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
celerycam_1   |   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
celerycam_1   |   File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
celerycam_1   |   File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
celerycam_1   |   File "<frozen importlib._bootstrap_external>", line 665, in exec_module
celerycam_1   |   File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/django_celery_monitor/camera.py", line 14, in <module>
celerycam_1   |     from .models import WorkerState, TaskState
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/django_celery_monitor/models.py", line 21, in <module>
celerycam_1   |     class WorkerState(models.Model):
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 105, in __new__
celerycam_1   |     app_config = apps.get_containing_app_config(module)
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 237, in get_containing_app_config
celerycam_1   |     self.check_apps_ready()
celerycam_1   |   File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 124, in check_apps_ready
celerycam_1   |     raise AppRegistryNotReady("Apps aren't loaded yet.")
celerycam_1   | django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

I am using:

Django==1.10.7
celery==4.0.2
django-celery-results==1.0.1
django-celery-beat==1.0.1
django-celery-monitor==1.0.1
anyjson==0.3.3

Any ideas why i am getting this or how to fix it??

THanks

When using django_celery_results I have to set this to django-db, but with this packege, when I am running a task and I am query-ing for it's state I am getting:

'DisabledBackend' object has no attribute '_get_task_meta_for'

In the docs it is stated that the name of the variables for the timedelta of expiration is

• monitor_task_pending_expires
• monitor_task_error_expires
• monitor_task_success_expires

but in the code it appears to me that the names are different

• monitors_expire_success
• monitors_expire_error
• monitors_expire_pending

Hi,

I am using django==1.8.19, celery==4.1.1,django-celery-monitor==1.1.2. And I want celery tasks to be preserved a bit longer in celery monitor. But, tasks stays only for 5 days max. I tried making configuration as stated in the readme. But, no use. Here is my code.

from datetime import timedelta
monitor_task_success_expires = timedelta(days=6*30)
monitor_task_error_expires = timedelta(days=6*30)
monitor_task_pending_expires = timedelta(days=6*30)`

All other settings in my django settings file are capital letters. do I have make these also in capital letters? like this?

MONITOR_TASK_SUCCESS_EXPIRES = timedelta(days=6*30)

please help me.

Hello,
I have this error:
ImportError: cannot import name 'render_to_response' from 'django.shortcuts'

when will it be compatible with django 3 and celery 5?

regards
Matias

Currently, calling a task with arguments and kwarguments results in something like this:

>>> task.args
[True]
>>> task.kwargs
{'arg1': 'some_string', 'arg2': False}

This is a bit of a pain when parsing this information, in particular to send it to a Javascript frontend, since it's almost JSON but not quite.

Could there be a setting to enable saving this information as properly encoded JSON? e.g.

>>> task.args
[true]
>>> task.kwargs
{"arg1": "some_string", "arg2": false}

Hi there,

I'm using the following decorator for one of my tasks:

@app.task(autoretry_for=(Exception,))

I was hoping that the number of retries would be reflected on the "retries" column under table celery_monitor_taskstate but for some reason, it's always 0. I'm positive that the task is failing and retrying since I can see the state changes via Flower.

Any ideas? Is there anything I'm missing?

It would be nice to have example scripts, similar to Celery itself, for upstart/systemd/etc. While I'm sure I can just copy and modify the celerybeat scripts for the purpose, I would be more confident if I could bring something over directly from this project.

Hello, it is said that we can control a few things:

• monitor_task_success_expires -- Defaults to timedelta(days=1) (1 day)
• monitor_task_error_expires -- Defaults to timedelta(days=3) (3 days)
• monitor_task_pending_expires -- Defaults to timedelta(days=5) (5 days)

Is this implemented ? Nothing works and does not appear in the source code ?

I have set them as followed in the settings.py file and didn't work at all...

monitor_task_success_expires = timedelta(seconds=10) # State: SUCCESS
monitor_task_error_expires   = timedelta(days=7) # State: RETRY, FAILURE, REVOKED
monitor_task_pending_expires = timedelta(days=7) # State: PENDING, RECEIVED, STARTED, REJECTED, RETRY

Could you provide an example of where to set this ?

Thanks a lot,

Jonathan

Is it possible to specify a name of a database that contains tables for the celery monitor? In my project it is not 'default'. My 'default' is empty.

I see that django-celery-monitor recently became a Jazzband project, but doesn't look like it's maintained.
It doesn't even work with Django 3.0 (probably even 2.2, judging by the issues).

I just wanted to check if there are plans on activating the project or is it abandoned?

Thanks!

This should move the time value to the interval field, so it's more in line with cron-like UX patterns, and the date of the first iteration kept in the start_date field.

Same thing for the end date and indicate if it's inclusive or exclusive.

Hello.

I am getting a quite high CPU usage from the celery monitor process, around 15% constantly, I'm not having a big load or queue so I can't really explain that (I process maybe like maximum 5 tasks per seconds in the worst case).

Do you have the same behaviour ? How can I troubleshoot that more ?

Here is my monitor command

pipenv run celery -A website_manager_service events -l info --camera django_celery_monitor.camera.Camera --frequency=2.0 --pidfile=

Thanks

Do you have any tips for migrating from celery 3.1.13 -> 4.x and django-celery 3.1.10 to [list the successor packages]? I'm in the process of migrating an old Django 1.6 site to Django 1.8. At the moment everything runs well under Django 1.8 with celery 3.1.25 and django-celery 3.2.1, but I want to migrate it a little bit further, keeping current functionality intact as much as possible.

Several parts of django-celery are currently in use (admin):

• /admin/djcelery/intervalschedule/
• /admin/djcelery/crontabschedule/
• /admin/djcelery/taskstate/
• /admin/djcelery/periodictask/
• /admin/djcelery/workerstate/

Any advice is highly appreciated. Thanks in advance!

Currently we're using Django's automatic transaction management when we call get_or_create or update_or_create. We should in addition use select_for_update to make sure that the data we're writing to the TaskState and WorkerState models work as intended.

This is especially true if the Camera is deployed in a way that would have multiple instances run side by side for scalability reasons.

Hi!

I am having an issue with some tasks that once they have successfuly finished do not update on admin panel, so they freeze with started or retry status.

I run celery monitor with:

celery -A proj events -l info --camera django_celery_monitor.camera.Camera

I didn't find anything in the logs for now.

Celery Monitor no longer works with Django 2.2:

More specifically:

[2019-04-05 10:05:57,937: ERROR/MainProcess] Error in timer: TypeError('get() argument after ** must be a mapping, not str')
Traceback (most recent call last):
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 166, in apply_entry
    entry()
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 127, in _reschedules
    return fun(*args, **kwargs)
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/celery/events/snapshot.py", line 72, in capture
    self.state.freeze_while(self.shutter, clear_after=self.clear_after)
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/celery/events/state.py", line 456, in freeze_while
    return fun(*args, **kwargs)
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/celery/events/snapshot.py", line 69, in shutter
    self.on_shutter(self.state)
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/camera.py", line 125, in on_shutter
    self.handle_worker(worker)
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/camera.py", line 77, in handle_worker
    update_freq=self.worker_update_freq,
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/managers.py", line 62, in update_heartbeat
    defaults={'last_heartbeat': heartbeat},
  File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/managers.py", line 33, in select_for_update_or_create
    obj = self.select_for_update().get(**lookup)
TypeError: get() argument after ** must be a mapping, not str

It looks like it's an issue where the django internal method _extract_model_params is no longer returning (lookup, params) but only params

Link to the django source code (search for def _extract_model_params):
https://docs.djangoproject.com/en/2.1/_modules/django/db/models/query/
https://docs.djangoproject.com/en/2.2/_modules/django/db/models/query/

When I use stopwait to stop running workers, it waits correctly, but the last task to run does not get its status updated to SUCCESS, it remains STARTED, even when workers and camera are restarted.

Start with your steps, and there is data in the wokerstate table, but there is no data in taskstate?

Your master build is failing, presumably due to the HTTP 503 error in https://travis-ci.org/jezdez/django-celery-monitor/jobs/324656984. It looks like a temporary network issue, so rerunning the build might do the trick.

Can you guys rerun the build?

We have many of this errors and produced in this code:

    def update_heartbeat(self, hostname, heartbeat, update_freq):
        with transaction.atomic():
            # check if there was an update in the last n seconds?
            interval = Now() - timedelta(seconds=update_freq)
            recent_worker_updates = self.filter(
                hostname=hostname,
                last_update__gte=interval,
            )
            if recent_worker_updates.exists():
                # if yes, get the latest update and move on
                obj = recent_worker_updates.get()
            else:
                # if no, update the worker state and move on
                obj, _ = self.select_for_update_or_create(
                    hostname=hostname,
                    defaults={'last_heartbeat': heartbeat},
                )
        return obj

We have identified the bug in interval line.

Interval is generated in dynamic way (celery_monitor_workerstate.last_update >= ((CURRENT_TIMESTAMP - INTERVAL '60.000000' SECOND_MICROSECOND))) and the query results from recent_worker_updates.exists() and obj = recent_worker_updates.get() is different.

With a fixed interval value, we can avoid this bug.

So currently HTML in the admin is not supported and HTML tags get rendered. Needs to be refactored to use format_html.

https://docs.djangoproject.com/en/2.0/ref/utils/#django.utils.html.format_html

Hey @jezdez
Does django-celery-monitor conflict with django-celery-results. I have a couple issues but I'm wondering if these two results are effectively stepping on each other.

Would you mind sharing how you have this setup and if you are using periodic tasks?

Thanks

Hello

I don't know you but I think it should be interesting to set the color of the STARTED string state in the admin to something darker, it's quite difficult to read this flashy yellow on a white background :)

Thanks

hi @jezdez i just realized that you are not "storing" the celery beat scheduled tasks. The old django-celery integration allowed this so we could keep track if the beat was running properly. so i was wondering if that was intentional or not.

I remember a while ago i asked for something similar here celery/django-celery-results#16 but maybe this might make sense in this project.

My recommendation was to save it to cache memory first and eventually dumping it in the database (could be the same table or a different table).

What do you think?

This issue tracks the implementation of the Jazzband guidelines for the project django-celery-monitor

It was initiated by @jezdez who was automatically assigned in addition to the Jazzband roadies.

See the TODO list below for the generally required tasks, but feel free to update it in case the project requires it.

Feel free to ping a Jazzband roadie if you have any question.

TODOs

• Fix all links in the docs (and README file etc) from old to new repo
• Add the Jazzband badge to the README file
• Add the Jazzband contributing guideline to the CONTRIBUTING.md or CONTRIBUTING.rst file
• Check if continuous testing works (e.g. Travis CI, CircleCI, AppVeyor, etc)
• Check if test coverage services work (e.g. Coveralls, Codecov, etc)
• Add jazzband account to PyPI project as maintainer role (e.g. URL: https://pypi.org/manage/project/django-celery-monitor/collaboration/)
• Add jazzband-bot as maintainer to the Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/django-celery-monitor/users/)
• Add incoming GitHub webhook integration to Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/django-celery-monitor/integrations/)
• Fix project URL in GitHub project description
• Review project if other services are used and port them to Jazzband
• Decide who is project lead for the project (if at all)
• Set up CI for Jazzband project releases if needed and open ticket if yes

Project details

can be closed, false positive

Dependabot can't evaluate your Python dependency files.

As a result, Dependabot couldn't check whether any of your dependencies are out-of-date.

The error Dependabot encountered was:

Illformed requirement [">dev"]

Hi,

I can't visualize meta information like root id, parent id, ...
This is normal?
With django_celery_results I have this informations

Thank

Why might the list of tasaks be empty? When I look at the log with tail I can see tasks beeing processed. But somehow these are not showing up for the monitor.

What will this package actually deliver? Extra commands? A webpage? An alternative to flower?