mayank1101 / crawlers

Web Crawler

Web Crawler (Scrapy)

These crawlers are developed as a part of Major Project CLOUD BASED SEARCH ENGINE. This crawler is developed basically to crawl different sections of StackExchange. Crawlers which are developed in this project crawls Artificial Intelligence, askUbuntu, and Astronomy section of the http://www.stackxchange.com

Pre-requisites

1. Python This project was developed on Ubuntu so Python was already install. For windows this link might be helpful http://www.howtogeek.com/197947/how-to-install-python-on-windows/
2. Install Scrapy https://doc.scrapy.org/en/latest/intro/install.html
3. Install virtualenv https://doc.scrapy.org/en/latest/intro/install.html#intro-using-virtualenv
4. SqlAlchemy This link will be helpful http://docs.sqlalchemy.org/en/latest/intro.html#installation-guide
5. PostgreSql These links will be helpful https://help.ubuntu.com/community/PostgreSQL https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-14-04