Penn State Software Engineering Capstone Project by Kurt Urban & Kofi Shang
Project Scrubber is a web application designed to automate and simplify the process of cleaning CSV files. Leveraging modern cloud technologies, it provides users with a robust online service where they can upload their data files, automatically correct common issues such as duplicates and formatting errors, and download the cleaned data.
- Frontend: React.js
- Backend: Node.js with Express.js
- Data Processing: AWS Lambda, Python, Pandas, DBSCAN
- Storage: AWS S3
- CI/CD: GitHub Actions, Pulumi
- Logging & Monitoring: AWS Lambda Powertools (Logger, Metrics, Tracer)
- Testing: Jest, React Testing Library, Pytest, SAM
To set up and run Project Scrubber locally, you need to have the following installed:
- Node.js (version 20 or higher)
- Yarn
- Python 3.11
- AWS CLI configured with appropriate credentials
- Pulumi CLI
- Pulumi CLI is used to deploy the AWS infrastructure. You can install it by following the instructions here.
- The Pulumi File used for this project is highly customized with various hard coded configurations. Should you wish to deploy the infrastructure, you will need to modify the Pulumi file to suit your needs.
-
Clone the repository:
git clone https://github.com/Kurt-Urban/Scrubber.git cd project-scrubber
-
Install dependencies:
yarn install
-
Start the frontend:
yarn start:frontend
-
Start the backend:
yarn watch:backend yarn start:backend
-
Open your browser and navigate to
http://localhost:3000
to view the app. -
Lambda Function:
Replace the file_key in event.json with a local file path and run the following command to test the lambda function.
cd packages/lambda pip install -r requirements.txt sam local invoke -e event.json