System Design
A simple sketch on system architecture (system-design.jpg)
Tip: This is an experimental project, So I decided to implement a simple Authentication layer to learn how libraries like OAuth works, But for sure we must use standard libraries in the production.
Best practices
- Project Structure, Grouping files by technical role (Configs, Routers, Models, Controllers)
- Develop your own common utilities as a NPM package (Rate limitters, ...)
- Separate Express App and Server
- Secure Environment Variables and Hierarchical Config
- Well defined Error Handling
- Eslint
- Naming-Tagging (Asset-Inventory)
- Caching
- Authentication Layer (OAuth, JWT)
- Automations Test
- Code Coverage
- CORS
- Filter Input, Escape Output
- Documentaion (includes API-DOC)
- Lock Dependencies (npm-shrinkwrap.json)
- Using Snyk.io
- Monitoring
- Logging (APM, ELK, ...)
- Exit the process Gracefully
- CI-CD
Project Structure
- GIT
- Syestem Design Sketch
- Dockerize (Node, MongoDB, Redis, ELK, RabbitMQ)
- PM2
- Env Variables
- package.json
- Levenshtein Distance Algorithm
- Query sanitizer
- Unit Test
- Functional Test
- File Manager (Local Disk, S3, ...)
- Nodemailer
Setup Third Party Services
- MongoDB
- Redis
- RabbitMQ
- ELK
Security checklist to consider
- Mongo Least Privilege Principle (sample: user with Read-Only access)
- Third-Party Packages (if there is a package with too many dependencies, you'd better write your package)
- Lock dependencies with npm shrinkwrap
- Protect User Data (encrypting DB, HTTPS, ...)
- Logging (failed logins, DB interactions, ...)
- Rotate keys (use services like AWS Secrets-Manager)
- Separate development and production environments (dockerize your development dependencies)
Resource Management
- Protect yourself from DoS Attacks (Rate-limiters, Debouncers, and Throttlers)
- Tweaking computing resources based on some custom benchmarks
- Make balance between CPU cores and allocated Memory
- Scalability
TO DO
- Login by Google and Outlook
- ....
Build and Start
sudo sysctl -w vm.max_map_count=262144
docker stop $(docker ps -aq); docker-compose up --build
# There will be 6 different containers:
# 1- backend-lengoo
# 2- mongodb-lengoo
# 3- mongoExpress-lengoo
# 4- elk-lengoo
# 5- redis-lengoo
# 6- rabbitmq-lengoo
# Log in to any container
# docker exec -it <container-name> bash
docker exec -it backend-lengoo bash
# Open a new terminal session to bottstrap system prerequisites
curl -XPOST http://127.0.0.1:8090/bootstrap
Run Eslint
docker exec -it backend-lengoo bash
npx eslint .
Run Automations Tests
docker exec -it backend-lengoo bash
./node_modules/mocha/bin/mocha --exit test/unit/index.js
./node_modules/mocha/bin/mocha --exit test/functional/index.js
Generate API documentation (api-doc.png)
docker exec -it backend-lengoo bash
apidoc -i app/controllers/ -o api-docs/
Generate code coverage report
docker exec -it backend-lengoo bash
nyc mocha test/unit/ --exit
nyc mocha test/functional/ --exit
Sample API Calls
# Login with admin user
curl -XPOST http://127.0.0.1:8080/users/login \
-H 'Content-Type: application/json' \
-H 'App-Version: 1.0.0' \
-H 'Scope: user' \
-H 'Client: webApp' \
-d \
'{
"email": "[email protected]",
"password": "md5(123456)"
}'
# Register a new user
curl -XPOST http://127.0.0.1:8080/users/register \
-H 'Content-Type: application/json' \
-H 'App-Version: 1.0.0' \
-H 'Scope: user' \
-H 'Client: webApp' -d \
'{
"displayName": "user-name",
"email": "[email protected]",
"password": "md5(123456)"
}'
# ==> Output:
# {
# "status": "success",
# "user": {
# "displayName": "user-name",
# "roles": ["USER"],
# "_id": "5f6c8d3a2bc09200148ed66c",
# "email": "[email protected]",
# "created": "2020-09-24T12:12:42.480Z",
# "updated": "2020-09-24T12:12:42.480Z"
# }
# }
# Login with email address
curl -XPOST http://127.0.0.1:8080/users/login \
-H 'Content-Type: application/json' \
-H 'App-Version: 1.0.0' \
-H 'Scope: user' \
-H 'Client: webApp' \
-d \
'{
"email": "[email protected]",
"password": "md5(123456)"
}'
# ==> Output:
# {
# "status": "success",
# "accessToken": "v9pn42a2qbggadowzus6fgh0",
# "refreshToken": "qawf66dxfvmi2mqy60z8orj6"
# }
# Upload a file
curl -XPOST http://127.0.0.1:8080/api/upload \
-H 'Authorization: v9pn42a2qbggadowzus6fgh0' \
-H 'App-Version: 1.0.0' \
-H 'Client: webApp' \
-H 'Scope: user' \
-F "fileName=Matrix-1999-en.txt" -F "sourceLanguage=en" -F "targetLanguage=de" -F "subtitle=@/home/saeed/subs/Matrix-1999-en.txt"
# Introduce translation data
For more sampling: [sample-data.md](resource/sample-data.md))
curl -XPOST http://127.0.0.1:8090/api/translations \
-H 'Authorization: <admin-token>' \
-H 'Content-Type: application/json' \
-H 'App-Version: 1.0.0' \
-H 'Scope: admin' \
-H 'Client: webApp' -d \
'[
{
"source": "Hello World",
"target": "Hallo Welt",
"sourceLanguage": "en",
"targetLanguage": "de"
},
{
"source": "Hello guys",
"target": "Hallo Leute",
"sourceLanguage": "en",
"targetLanguage": "de"
},
{
"source": "I walk to the supermarket",
"target": "Ich gehe zum Supermarkt.",
"sourceLanguage": "en",
"targetLanguage": "de"
},
{
"source": "I am Arwen - Ive come to help you",
"target": "Ich bin Arwen - Ich bin gekommen, um dir zu helfen.",
"sourceLanguage": "en",
"targetLanguage": "de"
},
{
"source": "Come back to the light.",
"target": "Komm zurück zum Licht.",
"sourceLanguage": "en",
"targetLanguage": "de"
},
{
"source": "Nooo, my precious!!.",
"target": "Nein, my Schatz!!.",
"sourceLanguage": "en",
"targetLanguage": "de"
},
{
"source": "Nooo00, my precious!!.",
"target": "Nein, my Schatz!!.",
"sourceLanguage": "en",
"targetLanguage": "de"
},
{
"source": "Nooo, dear precious!!.",
"target": "Nein, my Schatz!!.",
"sourceLanguage": "en",
"targetLanguage": "de"
}
]'