ammfat / faang-serverless-etl

Serverless ETL for FAANG stocks from RapidAPI 's Bloomberg API to BigQuery.

Serverless ETL for FAANG Stocks

Serverless ETL for FAANG stocks from Bloomberg API to BigQuery.

Data source: RapidAPI Bloomberg

How to Run (local)

1. Build a virtual environment.

   python -m venv venv
   source venv/bin/activate

2. Install the dependencies.

   pip install -r requirements.txt

3. Initiate environment variables in .env file.

   e.g.:

   export RAPIDAPI_API_KEY=Your_RapidAPI_key
   export BQ_PROJECT_DATASET_TABLE=Your_BQ_table_id
   echo -e \
   "RAPIDAPI_API_KEY=\"$RAPIDAPI_API_KEY\"\nBQ_PROJECT_DATASET_TABLE=\"$BQ_PROJECT_DATASET_TABLE\"" \
   > .env

   Look at .env_sample as environment file reference.

4. Run the application

   python app.py

   Example output:

   (venv)$ python app.py
   Invoke
   Extracting data from bloomberg api...
   Transforming data from bloomberg api to dataframe...
   6 new record(s) are found
   New rows have been added.
   6 record(s) inserted to BigQuery Table
   
   (venv)$ 
   

5. Check the BigQuery table and ensure the data successfully loaded.

How to Run (Serverlessly)

Refer to this Medium article to deploy it on GCP.