davidycliao / biscrawler

An Automation Webcrawler for Extracting Central Bankers' Speeches

bisCrawler: An Automation Webcrawler for Extracting Central Bankers' Speeches 🛠️🧰

An automation web crawling framework for retrieving for Extracting Central Bankers' Speeches on the Website of Bank for International Settlements (https://www.bis.org) based on Selenium and Chrome browser.

Environment Setup

1. Need to install Anaconda Navigator and Python>=3.9 beforehand. And then, open the terminal and download bisCrawler repository by using git clone. About how to use git and Github, please have a look at this Tutorial for Beginners.

git clone  [email protected]:davidycliao/bisCrawler.git

2. Copy the commands below and paste them into the terminal:

# Change the directory by typing `cd` command once `bisCrawler` repository is downloaded.
cd bisCrawler

# Create the enviroment by using conda and name the enviroment `bisCrawler`.
conda create -n bisCrawler python=3.9

Instruction

1. Activate the pre-named enviroment. Alternatively, the environment for bisCrawler can be opened via Anaconda Navigator

conda activate bisCrawler 

2. Install the dependencies from requirements.txt using pip methond.

pip install -r requirements.txt   

3. Call bisCrawler Moduel

• In the terminal:

# Note: you need to run it in the terminal where you activated the enviroment.
python bisCrawler.py

• In Jupyter Notebook:

from bisCrawler import scraper 

scraper()

4. When Running bisCrawler

When bisCrawler is running, you will be asked which page you would like to scrape (please, type any single digit from 1 to last page). Then bisCrawler will automatically generate pandas dataframe to restore the banker speeches and the urls to the textual document.

What bisCrawler Scrapes

This designed crawler automatically webscrapes the central bankers' speeches from the offical website, including a bunch of information with regards to each name of central banker, date and title and corresponding url to the textual document.

Websraped Data

The scraped dataframe will be stored as central_bank_speeches.csv in the bisCrawler folder.

Cite

Please cite this page if you use this toolkit for your research.

For example, with BibTeX:

@misc{bisCrawler,
    howpublished = {\url{https://github.com/davidycliao/bisCrawler}},
    title = {bisCrawler: An Automation Webcrawler for Extracting Central Bankers' Speeches},
    author = {David Yen-Chieh Liao and Li Tang},
    publisher = {GitHub},
    year = {2021}
}