A minimal, concurrent web scraper for Python
itsybitsy
features two implementations of the same crawling algorithm: One based
on asyncio
and aiohttp
that is only compatible with Python 3.5 and above, and a
requests
-based, multithreaded implementation that is supported by most other Python
versions. The requirements for itsybitsy
are thus dependent on which Python version
you intend to use.
For the asynchronous version, you will have to install aiohttp
, async_generator
,
and lxml
, e.g. via
pip install aiohttp lxml async_generator
It is also recommended to install cchardet
and aiodns
for increased performance:
pip install aiodns cchardet
For the multithreaded version, you need to install requests
and lxml
via
pip install requests lxml
itsibitsy
provides a single function, crawl
. The asynchronous version of the
crawler also supplies a coroutine called crawl_async
that you can use inside
async for
from other coroutines.
itsybitsy
contains a modified version of url_normalize
by Nikolay Panov, published under the PSF license.