mailbadger / app Goto Github PK

Is your feature request related to a problem? Please describe.
Currently, the templates that are used to send e-mails, are being stored in AWS as part of their SES templates feature, and we use the AWS api to create, update, delete and list templates.

The code for the CRUD operations reside in the storage/templates package, we'll want to move that to the upper layer. Our API endpoints for the templates resource should remain roughly the same, except maybe the List operation, we should adapt it to be the same as the rest of the paginated resources.

The templates that are created, are being used as part of the campaign operation. When we create a campaign we associate a template, to be rendered as an e-mail message when we send it to each subscriber. Templates are used as a way to send personalized e-mail to the customers, the subject and body can contain unique values that are personalized for each recipient.

Describe the solution you'd like

Let's start with the storage. A template consists of:

• A name
• Subject
• Text
• HTML

Text and html parts are the actual body, we can have an html template or send just a plain text message. Here are some example values:

{
    "name": "MyTemplate",
    "subject": "Greetings, {{.name}}!",
    "html": "<h1>Hello {{.name}},</h1><p>Your favorite animal is {{.favoriteanimal}}.</p>",
    "text": "Dear {{.name}},\r\nYour favorite animal is {{.favoriteanimal}}."
}

The html and text part must be a valid golang template i.e. parsable by the text/template package.

The table should look something like this:

The HTML part will be stored on s3 instead of the database since it's more suiting that way. We'll figure out the name of the bucket and folder later on. In case the user wants to send only a text email, not a fancy HTML one, we can skip that part.

Next up is the storage interface and implementation. Currently, this resides in the storage/templates, we should refactor this into our existing Storage interface, and re-implement all CRUD operations with the new store methods. We should also include tests since we don't have them because currently, it is a simple AWS API wrapper.

The final step, and probably the hardest one, is to refactor the send operation to use the templates from our end instead of the SendTemplatedEmail and SendBulkTemplatedEmail APIs.

Campaigns should be preserved as artifacts, since they are used in many places as foreign keys etc.

API endpoint

/api/campaigns/{id}/opens

Response object

{
  "total": 123,
  "per_page": 10,
  "links": {
    "previous": null,
    "next": "/api/campaigns/{id}/opens?per_page=10&starting_after=11"
  },
  "collection": [
    {
      "id": 10,
      "campaign_id": 5,
      "user_agent": "Android",
      "ip_address": "127.0.0.1",
      "recipient": "[email protected]",
      "created_at": "2020-08-08T23:00:0000Z"
    }
  ]
}

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/gin-gonic/[email protected] requires
	gopkg.in/[email protected] requires
	gopkg.in/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/go/gopath/pkg/mod/cache/vcs/9241c28341fcedca6a799ab7a465dd6924dc5d94044cbfabb75778817250adfc: exit status 128:
	fatal: The remote end hung up unexpectedly

API endpoint

/api/campaigns/{id}/bounces

Response object

{
  "total": 123,
  "per_page": 10,
  "links": {
    "previous": null,
    "next": "/api/campaigns/{id}/bounces?per_page=10&starting_after=11"
  },
  "collection": [
    {
      "id": 10,
      "campaign_id": 5,
      "recipient": "[email protected]",
      "type": "Permanent",
      "sub_type": "Suppressed",
      "action": "failed",
      "status": "5.0.0",
      "diagnostic_code": "X-Postfix; unknown user",
      "feedbac_id": "examplecode",
      "created_at": "2020-08-09 00:00:0000"
    }
  ]
}

The client should be able to consume the API endpoints with his api key.

API endpoint

/api/campaigns/{id}/complaints

Response object

{
  "total": 123,
  "per_page": 10,
  "links": {
    "previous": null,
    "next": "/api/campaigns/{id}/complaints?per_page=10&starting_after=11"
  },
  "collection": [
    {
      "id": 10,
      "campaign_id": 5,
      "user_agent": "Android",
      "recipient": "[email protected]",
      "type": "abuse",
      "feedback_id": "examplecode",
      "created_at": "2020-08-08T23:00:0000Z"
    }
  ]
}

We'll need to add user input validation on all of our action endpoints in a standardized way.

Use gin's model binding and validation as described here: https://github.com/gin-gonic/gin#model-binding-and-validation

Hello, this app seems nice but you must learn to code properly, without double code, comment one very function/method and use up to date design patterns.

Is your feature request related to a problem? Please describe.
We need a way to filter out collections based on different criteria:

• name
• template_name

The search results should be returned as a filtered out collection, the response will remain the same.

Describe the solution you'd like

API Example:

/api/campaigns?name=Foo&template_name=bar

The query params names will be the same as the fields that we are searching on. The query condition should be done with a LIKE where the wildcard % to be placed on the end. Example:

SELECT * FROM campaigns WHERE `name` LIKE 'Foo%'...

If the user sends multiple search filters (name and template_name), then the query expression should include an AND operator, so the filter will be narrowed down further.

We'll require one-time tokens to be used when a user wants to reset his password, or when a user wants to unsubscribe from a mailing list. The tokens will be stored in the mysql database and will have a custom expiration date.

This feature will require a new DB table with the following columns:

+------------+--------------+----------+-------------------+
|    Name    |     Type     | Nullable |       Extra       |
+------------+--------------+----------+-------------------+
| id         | integer      | NO       | auto_increment    |
| user_id    | integer      | NO       | references(users) |
| token      | varchar(191) | NO       | unique            |
| type       | varchar(191) | NO       |                   |
| expires_at | datetime     | NO       |                   |
| created_at | datetime     | NO       |                   |
| updated_at | datetime     | NO       |                   |
+------------+--------------+----------+-------------------+

For now, there will be only two types of tokens:

• verify_email
• forgot_password

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: honnef.co/go/[email protected]: unrecognized import path "honnef.co/go/tools" (https fetch: Get https://honnef.co/go/tools?go-get=1: EOF)

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

API endpoint

GET /api/campaigns/{id}/clicks

Response object

{
  "total": 123,
  "collection": [
    {
       "link": "http://example.com?foo=bar",
       "unique_clicks": 123,
       "total_clicks": 456
    },
    ...
  ]
}

Draft design of the flow:

Update the image if there are any changes, and add retries on this link

After the campaign is sent, we would want to see a report for our campaign. The report would have a total number of:

• sends
• deliveries
• opens
• clicks
• bounces
• complaints

This would be presented in the campaign's report component as a bar chart and also a percentage component, like:

API endpoint:

GET /api/campaigns/{id}/stats

Response object

{
  "recipients": 10000,
  "total_sent": 10000,
  "delivered": 5000,
  "opens": {
    "unique": 52,
    "total": 102
  },
  "clicks": {
    "unique": 25,
    "total": 32
  },
  "bounces": 456,
  "complaints": 128
}

API endpoint

/api/subscribers/{id}

Response object

{
  "id": 123,
  "name": "John",
  "email": "[email protected]",
  "metadata": {
    "foo": "bar"
  },
  "segments": [{
    // Segment details
  }],
  "blacklisted": false,
  "active": true,
  "created_at": "2020-10-01T15:00:00",
  "updated_at": "2020-10-01T15:00:00"
}

After a campaign is successfully sent, we can go to the campaign report screen to see a fully detailed report about the sent campaign. Here we'll show the campaign stats in a bar chart and a percentage stat component, as well as a table for each event that occurred (bounces, complaints, clicks, opens etc.).

Dependabot couldn't parse the go.mod found at /go.mod.

You can mention @dependabot in the comments below to contact the Dependabot team.

Role-based access control (RBAC) is a way of restricting system access to authorized users. There are several components to this system:

• Organization: an entity that holds the resources and comprises one or more users (subjects)
• Role: a job function or a title that defines some authority level
• Subject: a person or an agent that initiates operations
• Permissions: access mode for an operation or resource (read, write, publish, etc..)

The idea is to have a simple approach to access management that is based on assigning permissions to users based on their role within an organization. Instead of managing permissions individually, the users will conform to the permissions assigned to their role(s).

The plan is to use Open Policy Agent in order to enforce authorization policies. When the app needs to make a policy decision, we will defer to OPA and supply structured data as input. OPA then evaluates the query input against the defined RBAC policy and data and will return a decision (for example "allow" or "deny").

We will run OPA as a standalone service next to the API service and use the HTTP API to query it.

https://reactjs.org/docs/error-boundaries.html

Dependabot couldn't find a package.json for this project.

Dependabot requires a package.json to evaluate your project's current JavaScript dependencies. It had expected to find one at the path: /web/package.json.

If this isn't a JavaScript project, or if it is a library, you may wish to disable updates for it from within Dependabot.

You can mention @dependabot in the comments below to contact the Dependabot team.

A user with the admin role should be able to modify every resource and also be able to list all users.

Is your feature request related to a problem? Please describe.
We need a way to enable service limitations functionality on individual user basis. We envision four separate tiers:

• FOSS (free unlimited self-hosted tier)
• Free (free limited tier)
• Paid
• Enterprise

Based on the tiers there are certain limits on what a user can and can't do. For example:

• Free user is limited to:
  • X number of campaigns
  • X number of subscribers
  • Can't schedule campaigns
  • Can't create organizations

The full list of limits will be discussed separately, we just need an initial implementation for this feature.

Describe the solution you'd like
Let's start with the data storage first. We'll need two tables initially:

• tiers
• accounts_tiers (which account belongs to which tier)

Is your feature request related to a problem? Please describe.
We'll need a way of testing our action handlers. In this issue, we'll discuss and propose a solution.

Describe the solution you'd like
Choose a framework for testing our HTTP handlers and make one test that will serve as a blueprint for all future tests. Proposed frameworks are:

1. httpexpect
2. Ginkgo
3. testify

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/gobuffalo/packr/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	github.com/gobuffalo/[email protected] requires
	gopkg.in/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/go/gopath/pkg/mod/cache/vcs/9c8c6020ccff55230020a7f892faa28673bac3c13bb655b3edc1e9653672b90a: exit status 128:
	fatal: The remote end hung up unexpectedly

View the update logs.

Currently the campaign resource stores the template name in it's storage. After the templates refactor, the campaigns table should replace the template name with a foreign key to the templates table. The create and update operation will remain the same from the API perspective. Since the template name will be a unique value, we can search for the template by name and fetch the id and use that value when we create or update a campaign.

The get and list endpoints will be changed, to include a template object, instead of just the template name.
It should look something like this:

{
  "id": 123,
  "name": "Campaign Foo",
  "status": "draft",
  "template": {
    "id": 456,
    "name": "Foo",
    "subject": "Hello {{name}}",
    "created_at": "2020-12-17T12:00:0000",
    "updated_at": "2020-12-17T12:00:0000",
  },
   "created_at": "2020-12-17T12:00:0000",
   "updated_at": "2020-12-17T12:00:0000",  
}

Here are the steps we should take:

1. Remove template_name column
2. Add template_id as a foreign key column
3. Refactor Create and Update endpoints to fetch template by name and store the id in the campaign record.
4. Refactor Get and List to include template sub-object.

https://github.com/gorilla/csrf#setting-samesite

We should add the option for the client to delete his account and all of his subscribers and stats.

When sending promotional or newsletter content, we'll need to provide an unsubscribe functionality to our users.

When sending a bulk templated emails, for each subscriber we'll need to:

• Generate and sign a secure token using a secret.
• Create an unsubscribe URL with the given token as a query param.
• Add the unsubscribe url to the TemplateData to be used in the HTML template.

When the user asks to be unsubscribed from future emails (hitting the unsubscribe link), we'll need to:

• Grab the token from the query params.
• Call unsubscribe with the provided param.
• Present success/error message.

Endpoint

New endpoint(s) to schedule campaigns

Upsert endpoint

PATCH /campaigns/{id}/schedule

Request body:
schedule_date=2021-06-01 22:00:00

Response:
{
  "message": "Campaign <name> successfully scheduled at <date>"
}

Delete endpoint

DELETE /campaigns/{id}/schedule

Response:
{
  "message": "Campaign schedule removed successfully"
}

New table:

• For each campaign we can have only one record
• On campaign delete, cascade delete scheduled campaign record
• Boundary checks need to be performed to see if the user has access to this feature
• The scheduled_at should not be before now
• The campaign should have draft status

Start campaign changes

• If the user decides to start the campaign while it is scheduled, we should fetch the id from the schedule and send that as the event_id
• Update the campaign's status to sending and set the new column event_id from the scheduled record OR generate a new one in case the user has not scheduled the campaign beforehand

Campaigner changes

• Remove the status check as an idempotency measure
• Use the same method as the sender consumer to handle idempotency
• The send_logs record should contain this new event_id

Sender changes

• The idempotency key should be a combination of the event_id and subscriber_id
• The send_logs record should contain this new event_id

Campaign scheduler

We will have a cron scheduler that will fetch "expired" campaigns (scheduled_at < now) and publish message to the campaigner topic for all draft campaigns. We will set the event_id from the scheduled record's id

Is your feature request related to a problem? Please describe.
We need a way of exporting subscribers in a CSV file.

Describe the solution you'd like
We'll need an endpoint that will start the export process and send the CSV file to an S3 bucket. After the process is finished
the user will receive a pre-signed S3 URL so that he can use it to download the file directly.

Since the export process is a lengthy one, we'll do it in a separate goroutine (or maybe another consumer) and immediately send a response back to the user. When the process is done, we'll notify the user to download his file (which will be located in the reports UI section).

We can keep a list of all exports and present them to our user in a list, so he can choose which file he downloads. When the user chooses a file to download, we'll need to pre-sign the S3 URL and send it back to the user so he can use it in order to download the file.

Additional context
I think that we'll need two or maybe three endpoints for this feature:

1. Export subscribers endpoint (creates & uploads a CSV file to S3)
2. List all subscriber export reports.
3. Sign S3 url used for download.

The SES send quota should be part of the AWS SES settings component. After a user has successfully set his ses keys, we should fetch and present the send quota.