apsl / django-yubin

Send e-mails asyncronously using Celery

django-yubin

Django Yubin allows you to create, send and manage emails in your Django projects. It follows the 12-factors app methodology.

Yubin means postal service in Japanese. Thanks @morenosan for the name.

How it works

For creating and composing emails, Yubin provides class-based views that use standard Django templates.

For sending and queuing emails, Yubin replaces the standard Django Email Backend with its own. Instead of sending emails synchronously trough a SMTP server, Yubin saves emails in your database (and optionally in a file storage) and sends them asynchronously using the Celery distributed task queue.

Advantages

• Create and compose emails reusing your code easily with class-based views.
• Your app can respond requests faster because other process/worker is managing the connection with the SMTP server for sending emails.
• Scale out easily adding more Celery workers.
• Emails are saved in the database, you can see, manage and enqueue them from the Django Admin.
• Optionally you can save only minimum data in the database and full emails in a different storage.
• Yubin provides settings to avoid sending emails during development.

You can read the full documentation at http://django-yubin.readthedocs.org/

django-yubin's Issues

Rate limiting

I've been using django-yubin for a while and it's a great addition. My challenge is sometimes my queue gets too many items in it and then I have a large amount of emails to send (all opt-in, no spamming). My email provider doesn't like it when the volume spikes and starts aggressively throttling and can even lock the account for a while. It would be really helpful if django-yubin had some way of limiting the number of emails per time period. So feed from the queue but if we've sent X emails in this cron cycle then stop sending emails until next cycle.

Is anything like this possible now or is it possible to add something like this?

I found an option, block_size, in the code which I didn't see in the docs. It's related but still sends all mail in the queue.

Unicode e-mail address + Py3

Something I ran into when working on the test-suite: unicode e-mail addresses where the address itself is unicode is not possible on Py3. I believe this to be something Django/Python itself should handle, and submitted issues to both of them:

https://code.djangoproject.com/ticket/25986 and http://bugs.python.org/issue25955

Currently the unicode e-mail address test is skipped, but it's probably worth a mention in the docs that unicode e-mail addresses are problematic. I think however, that Django itself won't let you enter a unicode EmailField on Py3.

New mode to not save emails in database

We detect there is no mode to prepare and send email "now" without store it in database.
There is a problem if you want to use yubin's backend to send emails with large attachments.
The attachments will be stored completly in database with a storage and performance problem.

I propose a new mode or a way to send a email "now" without store it in db.
Maybe "live" mode?

No CI for pushes/pull requests (Travis)

I see that there is no Travis integration. I could add this (set up the whole tox stack) so you can at least see if the build is failing for certain Python/Django versions or not.

What's the support policy? Which Python/Django versions? I'd suggest Py27, Py34 and Py35, Django 1.8 and Django 1.9. For the rest I would point to django-mailer-2, which is unsupported but so are the Python/Django versions lower than those specified.

Data migration when updating from 1.x to 2.x is very inefficient leading to memory errors

There is a data migration in 2.0 that updates all Messages with some attributes needed for the move to Celery.

This data migration iterates over ALL Message objects and ALL Log objects. Specifically Message objects can be very big and take up a lot of memory (content, html content, and attachments).

The Message.objects.all() could be replaced by Message.objects.iterator() that is definitely present in the required Django 4.2 version as of Django Yubin 2.0.

See:

django-yubin/django_yubin/migrations/0007_auto_20200319_1158.py

Line 22 in 30d37c5

for message in Message.objects.all():

It's probably even more efficient to use update-statements to change all attributes at once, instead of iterating over all entries.

Use case: We had a migration that ran for 10+ minutes, after it was aborted because we suspected an issue. On a separate test environment, we ran the migration again but this time it gave an out of memory error after 20 minutes. More memory and patience can of course solve this, but the query can definitely be more efficient to prevent this.

Please make a release from the current master-branch

Hi,

It has been 2 months since any changes except the Celery bump. Can you make a new release with this change?

Thanks in advance!

Create an send e-mail test mode

Create a TEST_MODE and TEST_EMAIL_ACCOUNT to replace receiver e-mail with the TEST_EMAIL_ACCOUNT one.

To support creating a single Email object with several addresses

It would be nice to support multiple addresses (either with to / cc / bcc fields) for the same Email.
Currently, a new Message object is created per address.

By allowing the same email to be sent to multiple addresses, we could reduce table space significantly.

Add Timezone and tests for Django 1.7

See:

APSL/django-mailer-2#2 from @mjtorn

Django 4.2 support and dropping support for old versions

Hi - we're quite a heavy user of django-yubin in many of our projects, which we are looking to upgrade to Django 4.2 LTS. Looking at the metadata of this package, support does not seem confirmed yet.

I'd like to make PR in the next few days/weeks to indeed confirm this and iron out any kinks that would prevent using this package on modern Python and Django versions. However, looking at the tox envlist, I'm seeing a huge list of supported environments, dating back to Python 2.7 still!

So, I'd like to propose dropping support for End of life Python and Django versions in the process - is this something you'd be open to?

django-mailviews is from Disqus, not Quora

https://github.com/disqus/django-mailviews

message enqueuing and sending is susceptible to a race condition

We're observing failures to send queued emails in our environments after upgrading to Yubin 2.0 which uses celery to send the messages. Our Sentry logging shows errors from the celery works that it can't find messages and that would be explained by a race condition: the message sending task is scheduled and picked up by a worker before the message record is committed to the database.

We have large blocks of processing/business logic that are wrapped in @transaction.atomic which supports this theory.

Looking at the code, I don't see any consideration of DB transactions:

django-yubin/django_yubin/models.py

Line 257 in 37739ec

tasks.send_email.delay(self.pk)

Instead, this should be more conservative - messages should only be attempted to send when they are committed to the DB:

from django.db import transaction

...
transaction.on_commit(
    lambda: tasks.send_email.delay(self.pk)
)

Support for Django 1.11

Django 1.10+ does not accept a Context any more, thus autoescape off might be needed in the templates.

For reference:
pinax/pinax-notifications#68 (comment)
https://github.com/pinax/pinax-notifications/pull/70/files

Admin action to resend emails

Sometimes It is usefull to resend an email. It is easy from a Python shell:

m = Message.objects.get(pk=3456)
qm = QueuedMessage(message=m)
qm.save()

but It would be nice to be able to do It from the admin with an admin action https://docs.djangoproject.com/en/1.9/ref/contrib/admin/actions/

pyzmail36 broken

The last setuptools it seems removes the use_2to3 command and makes all the packages that rely on it broken.

  ERROR: Command errored out with exit status 1:
   command: /home/aaloy/.local/share/virtualenvs/hotchair-K6_Gp-Ql/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-t6kb0257/pyzmail36_5140f3a3eb3444d290819b4b2ee07fc9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-t6kb0257/pyzmail36_5140f3a3eb3444d290819b4b2ee07fc9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-uts0mdko
       cwd: /tmp/pip-install-t6kb0257/pyzmail36_5140f3a3eb3444d290819b4b2ee07fc9/
  Complete output (2 lines):
  error in pyzmail36 setup command: use_2to3 is invalid.
  VERSION 1.0.3

Add a «send_test_email» to check connection parameters

When deploying a django application the usual way to test the email system is to generate a contact request. But this action requires going into the website, knowing the page and making clicks.

Add a «send_test_email» to send a simple email in order to check connection parameters. The default configuration should send a "hello world" like email to the admin address.

Django-yubin 0.3.0

Hi, I'm trying to upgrade my django-yubin version to 0.3.0 and Django 1.10, but "pip install django-yubin" is working with 0.2.3 version (and pip install -e git+http://github.com/APSL/django-yubin.git#egg=django-yubin , too). I'm using Python 3.5.
Thanks for your time.

Lock already in place. Exiting.

Im having a significant issue with the lock system.

Usually yubin works great for everything. However, sometimes the lock file does not get deleted properly when the program exits. I dont know if its because of abnormal termination or due to some other reason.

Whatever the cause - after that, yubin cant run anymore because it sees the old lock file. Since Im not getting emails, sometimes it takes me days or weeks to realize that there is a problem. This happens rarely but often enough - roughly every few months.

Obviously this is a really serious issue on a production system.

I see there is a finally: block where the lock is supposed to get released. I have no idea why its not getting there sometimes.

Im not so familiar with the system, so the only solution I can think of is to create a management command that can be run say once or twice a day that checks for a really old lock file and deletes it. For instance a lock file thats 4 hours old is probably not really supposed to be there.

Maybe there is a more elegant and correct solution but thats all I can come up with given my limited knowledge

Thanks

Daniel

UnicodeDecodeError

Traceback (most recent call last):
File "./manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/home/healersource/lib/python2.7/django/core/management/init.py", line 354, in execute_from_command_line
utility.execute()
File "/home/healersource/lib/python2.7/django/core/management/init.py", line 346, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/healersource/lib/python2.7/django/core/management/base.py", line 394, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/healersource/lib/python2.7/django/core/management/base.py", line 445, in execute
output = self.handle(*args, **options)
File "/home/healersource/webapps/hs/env/lib/python2.7/site-packages/django_yubin/management/commands/send_mail.py", line 56, in handle
message_limit=options['message_limit'])
File "/home/healersource/webapps/hs/env/lib/python2.7/site-packages/django_yubin/engine.py", line 113, in send_all
blacklist=blacklist)
File "/home/healersource/webapps/hs/env/lib/python2.7/site-packages/django_yubin/engine.py", line 207, in send_queued_message
smart_str(message.encoded_message).encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 559: ordinal not in range(128)

Message.get_email_message discards information

We are queuing instances of EmailMultiAlternatives using django-yubin. This e-mail message is properly converted to string form in the Message model, but parsing this back into an e-mail to actually send via SMTP (via the Message.get_email_message method) drops a bunch of information.

Specifically, we are doing:

mail = EmailMultiAlternatives(...)
mail.mixed_subtype = "related"  # results in `Content-Type: multipart/related;` header
image = MIMEImage(content)  # base 64 content
image.add_header("Content-ID", f"<{cid}>")  # results in the Content-ID header for the image data
mail.attach(image)

Yubin parses this again using mailparser in

django-yubin/django_yubin/models.py

Line 137 in 2a0d59a

email = Email(

but in the process the mixed_subtype is lost, the base64 content of the attachment and the Content-ID header.

We are relying on this functionality to inline QR codes in confirmation e-mails.

I think that at the minimum a complex test like this should be added and the assertion made that:

message = EmailMultiAlternatives(...)
good = message.message().as_string()
instance = Message(message_data=good)
assert instance.get_email_message().message().as_string() == good

edit: I noticed this also loses any additional headers we pass to our email message constructor and even the reply_to kwarg

Sending emails with long subject lines gives "BadHeaderError: Header values can't contain newlines"

Long header lines get new lines \n added to them by https://github.com/django/django/blob/main/django/core/mail/message.py#L74 when the message is converted to a string.

When parsing the message again (https://github.com/APSL/django-yubin/blob/master/django_yubin/models.py#L134) the headers will have these extra new lines.

Trying to create an email (https://github.com/APSL/django-yubin/blob/master/django_yubin/models.py#L159) from this parsed message will raise the error "BadHeaderError: Header values can't contain newlines".

Add a «date_sent» field to detect when the mail was sent

https://github.com/APSL/django-yubin/blob/master/django_yubin/models.py#L43

Currently the model only saves when the message was created but there is no way to check when it was sent.

New 1.0.0 version doesn't work in django < 2.0

File "C:\evalite\evaliteCO\ENV\lib\site-packages\django_yubin\admin.py", line 7, in
from django.urls import reverse
ImportError: No module named urls

Worked fine with versions 0.7 and below.

django-yubin don't work with Django 1.10

If you try to run any manage.py command (runserver for instance) Django crashes and dumps this error trace:

  File "/home/marc/virtualenvs/doploy/lib/python3.5/site-packages/django_yubin/admin.py", line 13, in <module>
    from django.conf.urls.defaults import *
ImportError: No module named 'django.conf.urls.defaults'

Availability for send mails in a task system

Leaving the dependecy of execute django commands in a cron system.