Serverless ETL for FAANG stocks from Bloomberg API to BigQuery.
Data source: RapidAPI Bloomberg
-
Build a virtual environment.
python -m venv venv source venv/bin/activate
-
Install the dependencies.
pip install -r requirements.txt
-
Initiate environment variables in
.env
file.e.g.:
export RAPIDAPI_API_KEY=Your_RapidAPI_key export BQ_PROJECT_DATASET_TABLE=Your_BQ_table_id echo -e \ "RAPIDAPI_API_KEY=\"$RAPIDAPI_API_KEY\"\nBQ_PROJECT_DATASET_TABLE=\"$BQ_PROJECT_DATASET_TABLE\"" \ > .env
Look at
.env_sample
as environment file reference. -
Run the application
python app.py
Example output:
(venv)$ python app.py Invoke Extracting data from bloomberg api... Transforming data from bloomberg api to dataframe... 6 new record(s) are found New rows have been added. 6 record(s) inserted to BigQuery Table (venv)$
-
Check the BigQuery table and ensure the data successfully loaded.
Refer to this Medium article to deploy it on GCP.