Use create-virtualenv.sh to create a virtual environment for Python (needs PIP).

To generate the bots.txt file, you can use user-agents.sh script.

To convert the Apache log files to a BerkelyDB representation, use log2db.sh. After that you need to convert this in a BerkeleyDB queue for fast data analysis using the analyze.sh script.

The IPython notebook Group and tables.ipynb contains all the cells used to make the proper analysis of the data.