This repository contains code and documentation for a Big Data project completed during a master's program. The project involves data extraction using SQL, MapReduce, SparkSQL, and Parquet, as well as machine learning tasks such as text classification.
A detailed report of the EDA can be found in the big_data_report.pdf file.
- Implement various data extraction methods using SQL, MapReduce, SparkSQL, and Parquet.
- Compare execution times for different extraction methods.
- Test optimizers for merging data.
- Build a text classifier using TF-IDF technique.
A cluster master-slave setup has been deployed in Oceanos for data processing and analysis.