A simple web crawler in python
Requires python 2.x
No extra module needed
To test it , just run the crawl.py
example : python crawl.py
It will then automatically crawl pages as
long as its not stopped forcefully. And if
restarted later , it will start from just where
it stopped .
Pages crawled will be stored in data folder.
For storage issues , only the outgoing links
are stored in this case.