Patan is a lightweight web crawling framework, used to crawl website pages and extract data from the pages. It can be first helpful tool for data analysis or data mining. The core idea of Patan is inspired by Scrapy
- Python 3.7+
- linter: flake8
- formatter: yapf
- Lightweight: pretty easy to learn and get started
- Fast: powered by asyncio and multiprocessing(TBD)
- Extensible: both spider and downloader is designed to be opened for custom middlewares
pip install patan
patan newproject projectname
cd projectname
patan newspider spidername
start_urls = ['http://xxxx.html']
or you can override the start_requests method
def start_requests(self):
url_tpl = 'http://xxxx?page={}'
for i in range(1, 200):
url = url_tpl.format(i)
yield Request(url=url, callback=self.parse, encoding=self.encoding)
@dataclass
class StockItem:
field1: str
field2: str
# fieldn: type
class EastmoneyPipeline:
def process_item(self, item, spider):
# save item to database or files
logger.info(item)
{
"pipelines": {
"eastmoney.pipelines.EastmoneyPipeline": 10
}
}
{
"spider":{
"middlewares": {
"eastmoney.middlewares.EastmoneySpiderMiddleware": 200
},
},
"downloader":{
"middlewares": {
"eastmoney.middlewares.EastmoneyDownloaderMiddleware": 200
}
}
}
patan start [projectname|path-to-project]
- Settings File
- Middlewares
- Exception Handling
- Throttle Control
- Item Pipelines
- Scaffolding CLI
- Multiprocessing
- Pause and Resume
- Statistics Data Collecting
- Web UI
- More Protocols Support