kjam / python-web-scraping-tutorial

A Python-based web and data scraping tutorial

PyCon Introduction to Web and Data Scraping Tutorial

A tutorial-based introduction to web scraping with Python.

Virtual Env

If you'd like to use virtual environments, please follow the following instructions. It is not required for the tutorial but may be helpful.

For more details on virtual environments

If you don't have virtual env wrapper and/or pip:

$ easy_install pip
$ pip install virtualenvwrapper

and read the additional instructions here

$ mkvirtualenv scraper_tutorial
$ pip install -r requirements.txt

LXML and Selenium

You will need both LXML and Selenium to follow this tutorial in it's entirety.

If you are using a Mac, I would highly recommend using Homebrew. It will help make pip install very easy for you to use.

• Homebrew and LXML Installation
• More help on Installing LXML on Mac
• And additional suggestions for LXML on Mac

If you are using Windows, it might be worth it to run this within a Linux Virtual Machine. If you are a Windows + Python guru, please follow these installation instructions. I can help as needed but I have not programmed on Windows in more than 5 years.

• Installing Selenium on Windows
• Installing LXML on Windows

Please reach out to me if you have any questions on getting the initial requirements set up. Thanks!

Firefox Web Browser

Firefox comes as the default web driver for Selenium. To use Selenium easily, please download and install Firefox.

Using PIP

If you have never used PIP before you will need to sudo easy_install pip or brew install pip. PIP is a python package manager and it's really super so I highly advise using it!

Questions?

/msg kjam on freenode or @kjam on twitter

python-web-scraping-tutorial's Issues

Python 3 version

I should make this Python 3 compatible and update the code / libraries used, as it seems people enjoy it! 👍

example.webscraping is blocking ip address

• "TOO MANY REQUESTS" after the 12th request
• blocks ip address after five "TOO MANY REQUESTS" errors ...