patkle / cookiecutter-scrapy-project

Cookiecutter Template for Scapy Projects

Cookiecutter Scrapy Project

Features

• Configurations for Scrapy Cloud
• Configurations for Proxy Services:
  • Zyte Smart Proxy Manager
  • ScraperAPI

Requirements

• Cookiecutter for creating the project
• Scrapy for running the spider locally
• Shub if you want to deploy to Scrapy Cloud

Options

option	default	description
project_name	Scrapy Boilerplate	basically the root folder name
spider_name	Scrapy Spider Example	class name of the spider
proxy_service	Zyte Smart Proxy Manager	the proxy service used
deploy_to_scrapy_cloud	y	whether or not the project is to be deployed on scrapy cloud
use_spidermon	y	whether or not spidermon should be used for monitoring (and sending messages via telegram)
host_on_kaggle	y	whether or not the dataset is hosted on kaggle.com

ToDos:

• Add Spidermon
  • Telegram configs & templates
  • Some basic monitors
• Maybe add Scrapy Poet?

Also,

You could buy me a coffe if you wanted to. I'd really appreciate that.