mjrulesamrat / plaid_rest_celery

Plaid APIs integration with Django REST and Celery

Plaid API with Django Rest & Celery

"Few crazy out there who are willing to change the world will change it"

TLDR;

git clone [email protected]:mjrulesamrat/plaid_rest_celery.git

cd plaid_rest_celery

chmod +x app/entrypoint.sh

# most of default should work fine for quick demo except plaid keys & sentry
cp .env.example .env

cp plaid_rest_celery/settings/example-staging.py plaid_rest_celery/settings/staging.py

docker-compose up -d --build

docker-compose exec web python manage.py migrate --noinput

docker-compose exec web python manage.py createsuperuser

docker-compose exec web python manage.py collectstatic --noinput

# Cleanup:
docker-compose down -v  # down and remove volumes

Local Installation Guideline:

• Clone repository

git clone [email protected]:mjrulesamrat/plaid_rest_celery.git

• go to project directory

cd plaid_rest_celery

• Create Virtual Environment and install dependencies with Pipenv

pipenv install

• Activate virtualenv

pipenv shell

• Setup settings file

cp plaid_rest_celery/settings/example-staging.py plaid_rest_celery/settings/staging.py

cp .env.example .env

# change user and password at .env.rabbitmq

• Run migrations

python manage.py migrate

• Run project locally

python manage.py runserver

Admin panel

localhost:8000/plaid-admin/

Flower monitoring(Tested within docker)

• Basic auth creds, admin/admin or foo/bar

http://localhost:7775/

API docs using drf-yasg

• You'll need to login to admin panel first to access API docs

localhost:8000/api/v1/docs/

Celery local setup

• Start rabbitmq docker (Change user and password at .env.rabbitmq)

docker-compose -f docker-compose.rabbitmq.yml up -d

# If connection is refused despite right credenials. Troubleshoot with below commands.
docker exec -it container_id sh
rabbitmqctl list_users | grep user
rabbitmqctl change_password user password  # reset password

• start celery workers (Three workers with different queues)

celery -A plaid_rest_celery worker -Q flash -c 4 --loglevel=info
celery -A plaid_rest_celery worker -Q default -c 2 --loglevel=info
celery -A plaid_rest_celery worker -Q slow -c 2 --loglevel=info
celery flower -A plaid_rest_celery --address=127.0.0.1 --port=7775

Update requirements

Update requirements from Pipfile to requirements.txt. Everytime we do pipenv install, make sure to update requirements.txt

pipenv lock -r > requirements.txt

Logging

• Go to logs README

Testing guidelines (Tested with already created user, login to logout)

1. Login(Token authentication)

• Get access-token

curl -X POST -H "Content-Type: application/json" -d '{"username": "admin", "password": "admin"}' http://localhost:7777/api/v1/auth/token/login/

• response (Use this token in further requests)

{"auth_token":"dddad1111Example_token8218132251b"}

2. Get public token of customer from plaid

• Get Public token by running sandbox mode plaid-python-quickstart

3. Create an Item

• Send public token to create new item (Add user's plaid account)

curl -X POST -H "Authorization: Token dddad1111Example_token8218132251b" -d '{"public_token": "token"}' http://localhost:7777/api/v1/plaid/create-item/

4. Let the magic happen!

• background celery tasks will fetch access_token and item_id

• Then, it'll fetch and save item metadata

• then, it'll fetch and save accounts for given item

• At last, it'll fetch last 30 days transactions data for given item

5. Get Accounts information

curl -X GET -H "Authorization: Token dddad1111Example_token8218132251b" http://localhost:7777/api/v1/plaid/accounts/

6. Get transactions

curl -X GET -H "Authorization: Token dddad1111Example_token8218132251b" http://localhost:7777/api/v1/plaid/transactions/

7. Logout

curl -X POST -H "Authorization: Token dddad1111Example_token8218132251b" http://localhost:7777/api/v1/auth/token/logout/

plaid_rest_celery's Issues

Prometheus monitoring

• Integrate django-prometheus
• Run dashboard with separate docker-compose like rabbitmq
• Figure out secure way to connect prometheus docker to streamed log of django application

Plaid Integration

Token exchange API: An authenticated user can submit a plaid public token that he gets post link integration.

• This public token is exchanged for access token on the backend.
  - This initiates an async job on the backend for fetching account and item metadata for the access token.

Webhooks

Expose a webhook for handling plaid transaction updates and fetch the transactions on receival of a webhook.

Plaid error handling

Error References:

• handling item errors

• Plaid Complete Error References

Integrate Web link to get public-token

Currently I'm getting public-token running plaid python quickstart.

Once it is integrated in this project, we can send webhook URLs as well.

https://plaid.com/docs/#plaid-tokens-public_token-access_token-or-asset_report_token

Celery Integration

Integrate celery, celery-results-backend.

Add instructions to run rabbitmq as well.

Celery tasks

Add tasks to

• fetch item metadata (Fast)
• fetch accounts data (Normal)
• transactions data (Slow)

Three Queues and Three workers.

Containerize Application

Add docker-compose to include below services:

• nginx
• rabbitmq
• django
• worker

Authentication APIs

General user should be able to register, login and logout with REST APIs on the platform.

Add API docs.

Transaction API

Expose an API endpoint for fetching all transaction and account data each for a user.

Integrate flower

• Flower is a web based tool for monitoring and administrating Celery clusters
• Add one flower container in docker-compose