isabella232 / proxy-middleware

Scrapy middleware that reads proxy config from settings

proxy-middleware

Scrapy middlewares that reads proxy config from settings.

Contents

• Install
• ProxyFromSettingsMiddleware
• ProxyOnlyTorMiddleware
• License

Install

pip install proxy-middleware

ProxyFromSettingsMiddleware

A middleware that sets proxy from settings file.

Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:

DOWNLOADER_MIDDLEWARES = {
   'proxy_middleware.ProxyFromSettingsMiddleware': 10,
   ...

Pass proxy config via HTTP_PROXY and HTTPS_PROXY settings variables. HTTPPROXY_AUTH_ENCODING is also respected:

scrapy crawl my-spider -s HTTP_PROXY=http://localhost:8118

ProxyOnlyTorMiddleware

A middleware that proxies only requests to TOR sites (domain ends with ".onion").

Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:

DOWNLOADER_MIDDLEWARES = {
   'proxy_middleware.ProxyOnlyTorMiddleware': 10,
   ...

Settings: HTTP_PROXY for an http proxy and HTTPS_PROXY for an https proxy.

License

License is MIT.

---