kushtrimh / tomorr

Application that notifies you by email, when a Spotify artist you follow on 'tomorr' releases a new song/album.

tomorr

Associated projects

Tomorr Terraform (for provisioning the resources on AWS) - https://github.com/kushtrimh/tomorr-terraform

The project

Tomorr is a service that will notify you by email when an artist you follow on tomorr has a new release. It uses Spotify behind the scenes to get the artist's data and its releases.

This project is mostly done for the purpose of learning new technologies, and will be updated with different technologies in the future. I will not be deploying this project for other people to use.

You need to generate a client id and client secret from Spotify in order to start the service. For more information about that, please check https://developer.spotify.com/dashboard/applications.

API

Searching for artists

GET /api/v1/artist/search

Parameters:
    name: The artist's name
    external: true/false flag that defines whether only the local database should be used or not

Response:
    {
        "artists": [
            {
                "id": "1Xyo4u8u745435MpatF05PJ",
                "name": "Artist Name 1"
            },
            {
                "id": "0Y5tJX7345656lOH1tJY",
                "name": "Artist Name 2"
            }
        ]
    }

Following an artist

POST /api/v1/artist/follow

Request:
    {
        "user": "[email protected]",
        "artistId": "0Y5tJX7345656lOH1tJY"
    }

Response status code:
    200 - Success
    400 - Bad request
    404 - Artist not found
    409 - Artist already followed

Running the project

Using Docker

Building the image

Run docker build -t <IDENTIFIER>/tomorr:<TAG> . inside the project directory to build the image.

Running the container

Run

docker container run -p 8098:8098 -d --name tomorr <IDENTIFIER>/tomorr:<TAG>

or

docker container run -p 8098:8098 -d --name tomorr -v /home/config/application.yml:/etc/tomorr/application.yml <IDENTIFIER>/tomorr:<TAG>

in case you want to use an externalized configuration file.

Running with docker-compose

When running with docker-compose, create an .env file in the project directory, and add the environment variables specified below.

TOMORR_EXTERNAL_CONFIG=<DIRECTORY-TO-EXTERNAL-CONFIG>/application.yml

The external application.yml properties should contain the properties to needed to run the application when using docker-compose. Please check external configuration for a ready-to-use configuration file for this case.

Environment variables

If you do not plan on using an external application.yml file, then you can use environment variables to configure the application.

Environment variable	Description	Type
TOMORR_PID_FILE	PID file location.	string
TOMORR_MAIL_HOST	Mail host.	string
TOMORR_MAIL_USERNAME	Mail username.	string
TOMORR_MAIL_PASSWORD	Mail password.	string
TOMORR_MAIL_AUTH	Enable authentication for mail.	boolean
TOMORR_MAIL_TLS	Enable TLS for mail.	boolean
TOMORR_MAIL_PORT	Mail port.	integer
TOMORR_MAIL_FROM	Address where the email is sent from.	string
TOMORR_DB_URL	Database URL.	string
TOMORR_DB_USERNAME	Database username.	string
TOMORR_DB_PASSWORD	Database password.	string
TOMORR_SPOTIFY_CLIENT_ID	Spotify client id.	string
TOMORR_SPOTIFY_CLIENT_SECRET	Spotify client secret.	string
TOMORR_REDIS_HOST	Redis host.	string
TOMORR_REDIS_PORT	Redis port.	integer
TOMORR_RABBITMQ_HOST	RabbitMQ host.	string
TOMORR_RABBITMQ_PORT	RabbitMQ port.	integer
TOMORR_RABBITMQ_USERNAME	RabbitMQ username.	string
TOMORR_RABBITMQ_PASSWORD	RabbitMQ password.	string
TOMORR_RABBITMQ_SSL	RabbitMQ SSL status.	boolean

External configuration

The application.yml below shows an example of an external configuration file, and it also serves as a ready-to-use external configuration when using docker-compose. Please keep in mind updating client-id and client-secret.

spring:
  pid:
    file: /var/run/tomorr.pid
  mail:
    host: mail
    port: 1025
    username:
    password:
    properties:
      mail:
        auth: false
        starttls:
          enable: false
          required: false
mail:
  from: [email protected]
datasource:
  url: jdbc:postgresql://postgres:5432/tomorr
  username: postgres
  password: tomorrpostgres
spotify:
  client-id: <SPOTIFY_CLIENT_ID>
  client-secret: <SPOTIFY_CLIENT_SECRET>
redis:
  host: redis
  port: 6379
rabbitmq:
  host: rabbitmq
  port: 5672
  username: guest
  password: guest
  use-ssl: false

tomorr's Issues

RabbitMQ configuration

Add RabbitMQ support, using the appropriate Spring library.

RabbitMQ host and port should be configurable using env variables

host: TOMORR_RABBITMQ_HOST
port: TOMORR_RABBITMQ_PORT

Add functionality to add new association between the user and the artist

New Rest API endpoint /v1/follow/artist,
Accept: application/json
Content-Type: application/json

Request sample
{ "artist": "artist-id-value", "user": "[email protected]" }

Successful response, empty response body, 200 status code returned.

Failed response, and an appropriate response code based on the error
{ "error": "error-description" }

New Spotify API request to query an artist.

When adding an artist check if it exists on database, or in cache, if not then query the artist in Spotify to check if it exists there. If the artist does not exist, do not create association, if it exists, insert it on DB + Cache, and create association with the user.
Checks if association exists between user and artist should be done as well, being that each user will not have a big number of artists that it follows, those lists should be user based, and saved in cache as well, updated each time when a new artist is associated with the user.

Request counters for API calls should be implemented and saved in cache, and limits should be set in application properties too, since this is the first issue where actual API calls will be done.
There should be a global limit, which represents the number of requests that can be sent to Spotify per minute.

Implement email client

Implement a email client, which will be used to sent emails for security purposes, and later for notifications as well.
The service should be configurable through application.properties.

Add services for artist and user entities

The title sums it up pretty well

Create new Spotify Web API client

New Spotify Web API Client should be created with basic functionality which supports

• Get an artist by name
• Get an artist by id
• Get an album by id
• Get tracks of an album

There should be support for asynchronous requests as well.

Cache new albums for which notifications are already sent

Albums on Spotify seem to be added sometimes twice, same album but one that contains the explicit language, and one that does not.
To prevent sending notifications for the same album twice, once an album notification is sent, its name should be hashed (MD5) and cached.

All entries should be saved for at least 6 hours as an experiment, and later this duration can be changed.

Albums should be cached as follows,

album name=The new tomorr album
hashed album name=f8017f096552f596cea908e0b14c5a45

Redis entry notifiedAlbum:f8017f096552f596cea908e0b14c5a45 = 1
The value will be set as 1 just as a placeholder, and it will not be used. The functionality to check if the email notification should be sent for an album should be implemented using the EXISTS command.

Album sync task executor implementation

Executors will receive tasks from RabbitMQ albums sync queue, send a request to the Spotify API, and then process the response based on the task type.

Sync task type

For sync task type we will check if the artist total album count matches the current retrieved total album count from the API.
If they match, then no new album has been released, so the response is ignored, if they do not match, then we need to find the new released albums for the artist.

newAlbums = totalAlbumsFromAPI - totalAlbumsFromProject

The newAlbums is the count of new albums for that artist, and an iteration will start for each album in response, to check if it matches any album saved in the project.

If there is no new album in the current response body, then the a new task will be created with type 'Continued Sync Task Type', with the URL to get the next number of albums for the artist. When the new album is found however, that album is saved in database and cache, associated with the artist, and the count is increased. Also a new notification email is sent at RabbitMQ email queue for each user that is associated with the artist.

Initial Sync Task Type

For initial sync task types, the data for each retrieved album is saved in the database, and the count is increased, after that if there are more albums for the same artist, the 'next' request URL is attached to a new Initial Sync Task Type and added to the Task Queue in Redis.

Continue Sync Task Type

Has the same logic as Sync Task Type except that the count of albums retrieved from the API and the counts of albums saved in the project is not checked.

Email notification to users

Receive notifications from the RabbitMQ email notifications queue, and send emails for each of the received values.

Album sync task creator implementation

Implement a new service, which should run every minute, and create tasks to sync albums of new artists.
Each artist will have a column, which is 'sync_id', which will be an identifier of the current syncing process. This identifier will be a UUID saved in Redis, and once all artists are synced, it will be re-generated.
Only artists with sync_id different from the current generated one will be retrieved, once an artist is retrieved, a new task will be created and inserted into the albums sync task queue.

After the task is created, the artist sync_id, will be set to the current generated sync id, so it does not get retrieved again on the next run.

The task creator service, should check the amount of artists to retrieve, based on the count of API requests that can be sent to Spotify API within that minute.

numberOfArtistsToRetrieve = globalSpotifyAPICount - countOfRequestsSentToSpotifyAPIWithinThisMinute

The task creator, should be with a delay e.g. 59 seconds, so it does not get run after the count is reset. It should be run just before the count is reset, so other Spotify API requests get counted appropriately.

Tasks should contain the URL which the API will be sent at, and also the type of the task.

Task creator will always create tasks with Sync task type

Album syncing task queues

Add new task queues in Redis, which should contain data about querying the albums.

Task queues will be used later by the 'Albums Sync Task Creator' and also by the consumers when there is pagination while querying the albums.

There will be three task types:

• Initial sync task type
• Sync task type
• Continued sync task type

Initial syncing when a new artist is inserted implementation

When a new artist is added, all it's albums should be retrieved and saved. There should be a set in cache as well, for the artists saved albums.

To retrieve the albums, there should be created a task for this artist, with the type Initial sync task type

When saved, the count of the total albums should be saved for the artist as well.

Album sync task consumer implementation

Album task consumer is a service which should run every seconds, and get a specific number of tasks from the task queue, and send them to RabbitMQ.

The number of tasks retrieved per second is = globalSpotifyAPIRequestLimit / 60

Add authentication and authorization support

Add authentication and authorization support using Spring Security.

Features to be implemented

• Register with a new account, using email and password, two types of roles, USER and ADMIN.
• Send verification email
• New functionality to return JWT token for user (Authorization Server part)
• Add support to authorize with JWT tokens
• Forgot password functionality

Logging with ELK stack

Configure the application to forward application logs to ELK stack.
Logback should be used as as the logging library.