aerendir / bundle-aws-ses-monitor

Symfony Bundle to manage AWS SES notifications through AWS SNS.

Serendipity HQ AWS SES Monitor Bundle

AWS SES Monitor Bundle automates the filtering of bounced and complained e-mails sent through AWS SES.

Supports:

Supports:

Current Status

Features

AWS SES can give you notifications about bounced or complained e-mails both via e-mail or via the AWS SNS service.

Managing these notifications it is possible to know to which e-mails we should not send further e-mails.

This bundle helps you setting up the automatic handling of notifications via SNS.

---

Do you like this bundle?
LEAVE A ★

or run
composer global require symfony/thanks && composer thanks
to say thank you to all libraries you use in your current project, this included!

---

How AWS SES Monitor Bundle integrates with AWS SNS

Using AWS SES Monitor Bundle you can, using the console of your Symfony's App, create in AWS Simple Notifications Service a topic for bounced emails and one for complained emails and automatically subscribe your app identity to that topics.

The bundle exposes some endpoints called by AWS Simple Email Service when an e-mail is bounced or complained. When those endpoints are called, AWS SES Monitor Bundle persists these emails in the database.

The SwiftMailer plugin included reads these e-mails and automatically disable the sending of e-mails to those addresses.

The entire procedure is automated by the bundle in a really simple and easy way.

Documentation

You can read how to install, configure, test and use AWS SES Monitor Bundle in the documentation.

---

Do you like this bundle?
LEAVE A ★

or run
composer global require symfony/thanks && composer thanks
to say thank you to all libraries you use in your current project, this included!

---

bundle-aws-ses-monitor's Issues

Verify the signature of the message sent from SNS

https://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html

Fix badges on README.md

As on

• Aerendir/bundle-stripe#30
• Aerendir/bundle-commands-queues#47

Only use yaml for config

As recommended by best practices: Service Format: YAML

Support the MailObject

The AWS SNS has a MailObject part.

Save it into the database.

See: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-contents.html#mail-object

Automatically create the Aws\Credentials\Credentials service if not already created

[aws:ses:monitor:subscribe] Make possible to configure topics with --no-interaction

This way is possible to call the command on each deploy to test if all works well.

Think better at this: the main goal is to be sure the integration with AWS SES works as expected each time the app is deployed.

For example, check the app can connect to AWS.

Make also possible to automatically subscribe all not already subscribed identities.

See also #42.

Autoregister entities

Instead of requiring them to be added to the Doctrine's configuration.

See also #44.

Remove CodeClimate

As on

• Aerendir/bundle-stripe#34

• Aerendir/bundle-commands-queues#51

• Remove badges from README.md

• Remove connection with CodeShip (on codeship and in GitHub)

Implement hard_blacklist_time

Move tests to Tests

As suggested by Best practices: Tests.

As on

• Aerendir/bundle-stripe#37
• Aerendir/bundle-commands-queues#53

Test with SF3.4 and SF4.1 and PHP 7.1 and 7.2

Configure Travis-ci to test with both versions of Symfony and PHP.

Implement also other suggestions from best practices: Continuous integration.

As on

• Aerendir/bundle-commands-queues#44
• Aerendir/bundle-stripe#24

Dispatch some events

Hey,

It would be nice to dispatch some events about bounces, complaints so that we can subscribe to make extra treatments in our app.

Regards

Reactivate static analysis of Tests (PHPStan)

This requires phpstan/phpstan#1203 to be solved.

Then reactivate also the tests oh Phan.

Add "Star this" to documentation

As on

• Aerendir/bundle-commands-queues#55
• Aerendir/bundle-stripe#39

[Doc] Protocol: "The protocol to use..." for what?

It is not clear in the documentation the purpose of the parameter.

Use symfony phpunit-bridge

As on

• Aerendir/bundle-commands-queues#56
• Aerendir/bundle-stripe#41

Test with PHPStan

As on

• Aerendir/bundle-stripe#32
• Aerendir/bundle-commands-queues#49

Integrate with SonataAdmin

Make possible to see relevant information directly from the app front-end, using SonataAdmin.

Move docs to Resources

As suggested by best practices: Documentation.

As on

• Aerendir/bundle-commands-queues#52
• Aerendir/bundle-stripe#36

[Doc] Example with ENVs

In the documentation, the examples of configuration should now use ENVs as this is the new way of setting parameters.

As on Aerendir/bundle-stripe#23

Secure the endpoint/controller

[ ] Add query string param that is used as password;
[ ] Check if it is possible to verify the request really comes from Amazon.

As on Aerendir/bundle-stripe#2

Test with Phan

As on

• Aerendir/bundle-commands-queues#48
• Aerendir/bundle-stripe#31

Rename "Deliveries.enabled" into "track"

The enabled param is not sufficiently clear: rename it to track to make clear its purpose is to enable tracking also of the delivered emails.

The same for complaints and bounces.

[aws:ses:monitor:subscribe] Check the topic is not already subscribed

Goal: avoid to subscribe the same topic from development and production.

Force the use of `dev` and `prod` in the name of topics

This will allow for more precise checks about subscriptions

Use Symfony's coding standars

As on

• Aerendir/bundle-commands-queues#54
• Aerendir/bundle-stripe#38

Webhook: immediately return response and then dispatch event

• https://stripe.com/docs/webhooks#best-practices
• http://stackoverflow.com/questions/35219537/symfony-run-code-after-response-was-sent/35219628#35219628

As on Aerendir/bundle-stripe#4

Create a command to send a test email

See also #41.

Use only one endpoint

Add to Symfony Flex

https://github.com/symfony/recipes-contrib

• Create a config file in config/packages/shq_aws_ses_monitor.yaml with this content:

# Aws Ses Monitor
shq_aws_ses_monitor:
    aws_config:
        credentials_service_name: 'Aws\Credentials\Credentials'
        region: '%env(AWS_REGION)%'
    bounces:
        topic:
            name: '%env(AWS_SES_TOPIC_BOUNCES)%'
            endpoint:
                protocol: '%env(APP_SCHEME)%'
                host: '%env(APP_HOST)%'
    complaints:
        topic:
            name: '%env(AWS_SES_TOPIC_COMPLAINTS)%'
            endpoint:
                protocol: '%env(APP_SCHEME)%'
                host: '%env(APP_HOST)%'
    deliveries:
        topic:
            name: '%env(AWS_SES_TOPIC_DELIVERIES)%'
            endpoint:
                protocol: '%env(APP_SCHEME)%'
                host: '%env(APP_HOST)%'

• Create a config file in config/packages/dev/shq_aws_ses_monitor.yaml with this content:

# Aws Ses Monitor
shq_aws_ses_monitor:
    bounces:
        topic:
            endpoint:
                protocol: '%env(NGROK_APP_SCHEME)%'
                host: '%env(NGROK_APP_HOST)%'
    complaints:
        topic:
            endpoint:
                protocol: '%env(NGROK_APP_SCHEME)%'
                host: '%env(NGROK_APP_HOST)%'
    deliveries:
        topic:
            endpoint:
                protocol: '%env(NGROK_APP_SCHEME)%'
                host: '%env(NGROK_APP_HOST)%'

• Add those variables to .env file:
  • NGROK_APP_SCHEME
  • NGROK_APP_HOST
  • AWS_SES_VERSION
  • AWS_SES_TOPIC_BOUNCES
  • AWS_SES_TOPIC_COMPLAINTS
  • AWS_SES_TOPIC_DELIVERIES

Other variables used, but not only by AwsSesMonitorBundle
- AWS_SNS_VERSION
- AWS_ACCESS_KEY_ID
- AWS_ACCESS_KEY_SECRET
- AWS_REGION

As on

• Aerendir/bundle-stripe#20
• Aerendir/bundle-commands-queues#39

Only allow to configure the scheme and host for the endpoint

Currently the bundle requires a configuration like this:

# Aws Ses Monitor
shq_aws_ses_monitor:
    aws_config:
        credentials_service_name: 'Aws\Credentials\Credentials'
        region: '%env(AWS_REGION)%'
    bounces:
        topic:
            name: '%env(AWS_SES_TOPIC_BOUNCES)%'
            endpoint:
                protocol: '%env(APP_SCHEME)%'
                host: '%env(APP_HOST)%'
    complaints:
        topic:
            name: '%env(AWS_SES_TOPIC_COMPLAINTS)%'
            endpoint:
                protocol: '%env(APP_SCHEME)%'
                host: '%env(APP_HOST)%'
    deliveries:
        topic:
            name: '%env(AWS_SES_TOPIC_DELIVERIES)%'
            endpoint:
                protocol: '%env(APP_SCHEME)%'
                host: '%env(APP_HOST)%'

This is verbose and not useful.

Maybe a configuration like this is better:

# Aws Ses Monitor
shq_aws_ses_monitor:
    aws_config:
        credentials_service_name: 'Aws\Credentials\Credentials'
        region: '%env(AWS_REGION)%'
    topics:
        endpoint:
            protocol: '%env(APP_SCHEME)%'
            host: '%env(APP_HOST)%'
    bounces:
        # Remove
        topic:
            name: '%env(AWS_SES_TOPIC_BOUNCES)%'
    complaints:
        # Remove
        topic:
            name: '%env(AWS_SES_TOPIC_COMPLAINTS)%'
    deliveries:
        # Remove
        topic:
            name: '%env(AWS_SES_TOPIC_DELIVERIES)%'
            endpoint:
                protocol: '%env(APP__OTHER_SCHEME)%'
                host: '%env(APP_OTHER_HOST)%'

This way is the configuration is way more simple.

[aws:ses:configure] Create the command

Goal: avoid to subscribe, for example, coommercio.com from trustback.me.

In development only email like [email protected] can be subscribed. Maybe can be useful to make this email configurable (dev.test_identity).

This requires #82

Todos:

• Configure on SES Identities configured in bundle:
  • Create the identity if doesn't already exist (SesClient::VerifyEmailAddress() and SesClient::VerifyDomainIdentity());
• Check the topic is not already subscribed
• Topic is subscribed to identities
• App is subscribed to topic
• Based on configuration, make possible to enable or disable the inclusion of original headers in the SNS notification (SetIdentityHeadersInNotificationsEnabled);
• Based on configuration, make possible to enable or disable the email feedback forwarding (SetIdentityFeedbackForwardingEnabled);
• Check differences for dev env and prod env: on dev should not be possible to subscribe to domain identities, but only to the email identity [email protected] (that has to be verified and work as expected to make possible to test the real sending of emails through SES in development). The test email can be configured: if changed, the debug command should highlight that the test email doesn't have the test mailbox.

This command will substitute the exsisting ones:

• aws:ses:monitor:setup:bounces-topic
• aws:ses:monitor:setup:complaints-topic
• aws:ses:monitor:setup:deliveries-topic

Rename "Model" to "Entity"

Support DeliveryNotifications

http://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-contents.html#delivery-object

[aws:ses:debug] Create the command

The command has to check that all is configured well.

This requires #82

Checks to do:

• Account

  • The account is enabled for sending emails;
  • There still is quota available;

• Identities: list all identities and for each one list:

  • Is Identity verified?
  • Is DKIM enabled?
  • Is DKIM verified?

Test Command: Automatically find the from email

The corresponding identity has to be already verified or can be also not still verified?

Remove the AwsClienFactory and create services when the bundle is initialized

Dispatch events on relevant events

Events that will be dispatched:

• Recipient filtered out (before sending);
• Recipient filtered out (after sending);
• Bounce notification from AWS SES;
• Complaint notification from AWS SES;
• Delivery notification from AWS SES;

Requested in #31

Create an `aws:ses:monitor:unsubscribe` command

The command is meant to unsubscribe a topic from a notification type.

Fix as much as possible Sensio Insights errors

As on

• Aerendir/bundle-commands-queues#50
• Aerendir/bundle-stripe#33

Add mapping directly to the Entity classes

See also #40.

As on Aerendir/bundle-commands-queues#46

Reactivate the check on schema in configuration.php

Rename protocol config param to scheme

Rename

• shq_aws_ses_monitor.bounces.topic.endpoint.protocol to shq_aws_ses_monitor.bounces.topic.endpoint.scheme
• shq_aws_ses_monitor.complaints.topic.endpoint.protocol to shq_aws_ses_monitor.complaints.topic.endpoint.scheme
• shq_aws_ses_monitor.deliveries.topic.endpoint.protocol to shq_aws_ses_monitor.deliveries.topic.endpoint.scheme

Add a plugin to respect the Max Send Rate

https://docs.aws.amazon.com/ses/latest/DeveloperGuide/manage-sending-limits.html?icmpid=docs_ses_console

Add sh commands

As on Aerendir/bundle-stripe#35

Move the LICENSE to root folder

As recommended by best practices: Directory structure.

Implement soft_blacklist_time

Handle UnsubscribeConfirmation from topics

https://docs.aws.amazon.com/sns/latest/dg/json-formats.html#http-header

x-amz-sns-message-type = UnsubscribeConfirmation

Relevant code in HandlerFactory::buildHandler().

custom ORM model manager

Two issues when working with a custom entity manager:

In: aws-ses-monitor-bundle/Command/SnsSetupCommandAbstract.php

line 90 needs to be changed from:
$this->getContainer()->get('doctrine.orm.default_entity_manager')->flush();

to:
$this->getContainer()->get('aws_ses_monitor.entity_manager')->flush();

and line 181 from:
$this->getContainer()->get('doctrine.orm.default_entity_manager')->persist($topic);

to:
$this->getContainer()->get('aws_ses_monitor.entity_manager')->persist($topic);

Thanks!

Email statuses duplicated entry

Hey,

I have been getting few exceptions about duplicate entry in email statuses table. Any ideas?

An exception occurred while executing 'INSERT INTO aws_ses_monitor_email_statuses (hard_bounces_count, soft_bounces_count, last_bounce_type, last_time_bounced, complaints_count, last_time_complained, deliveries_count, last_time_delivered, email_address) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)' with params [0, 0, null, null, 0, null, 1, "2017-08-21 11:01:16", "[email protected]"]: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "aws_ses_monitor_email_statuses_pkey" DETALHE: Key (email_address)=([email protected]) already exists.

I am running Sf 2.8 with "serendipity_hq/aws-ses-monitor-bundle": "^1.2"