anna-liepina / explore-sa-python

	master	heroku
tests
coverage

'Data Explorer' GraphQL back-end (Python)

This project is done to demonstrate my knowledge, which I learned recently as I try to get into the software development industry, I had a mentor to help me out. The project aims to parse UK government data on property sales, and perform statistical analysis, as well as link it with geology data, like addresses and latitude and longitude.

the aim is to build a scalable GraphQL backend, which can quickly return requested results to demonstrate complex cases of GraphQL use, such as N+1 problem, scaling where more than one database is required [write/read nodes] complex automated QA, anonymized data seeding for QA purposes, and some limits of JavaScript, where for example by default objects in V8 object can have ~8.4mil of fields, but Map can handle way more. Queue system for data processing.

• GraphQL live demo [currently unavailable]
• Web Application example of how data can be consumed
  • Web Application live demo [currently unavailable]

software requirements

• Python v3.11+

used technologies

• fast API
• strawberry
• peewee
• graphql

where to get data-sets

• UK postcodes
• UK house sales data
• UK LSOA & Postcode data
• Police Records Data

how to install

• pip install -r requirements.txt

how to run in 'development' mode

• uvicorn main:app --reload

how to run in 'production' mode

• gunicorn --worker-class uvicorn.workers.UvicornWorker main:app
  • to run on different port example: PORT=18080 gunicorn --worker-class uvicorn.workers.UvicornWorker main:app

gitflow

• heroku -> current production, contains production specific changes, trigger production deploment on every push
• master -> most upto date production ready, all pull requests in to this branch got mandatory check 'ci/circleci: jest'
• feature branches -> get merged into the master branch when they are ready and mandatory checks passed
• CI executes tests in an isolated environment

used environment variables

variable	default value	used as	purpose
PORT	8081	number	port on which application will be made available
***	***	***
DB_HOSTNAME	127.0.0.1	string	host on which database can be reached
DB_USERNAME	root	string	database user
DB_PASSWORD	password	string	database user's password
DB_PORT	3306	number	port on which database can be reached
DB_NAME	explore	string	database [schema] name
DB_DIALECT	mysql	string	database's dialect: one of mysql / sqlite / postgres

supported databases

database	version	adapter	main purpose
MySQL	8	mysql2	production

• if you use MySQL 5.7+ you need to make sure it can work with mysql native password

• PostrgeSQL and SQLite are partially supported because some of the queries are not fully engine-agnostic, and some functions do not exist in SQLite for example