lstnull / scrapydweb

爬虫部署Scrapyd cluster management, Scrapy log analysis & visualization, Basic auth, Auto eggifying, Email notice. GIF DEMO:

ScrapydWeb: Full-featured web UI for Scrapyd cluster management, Scrapy log analysis & visualization

Features

• Scrapyd Cluster Management

  • Group, filter and select any numbers of nodes
  • Execute command on multinodes with one click

• Scrapy Log Analysis

  • Stats collection
  • Progress visualization
  • Logs categorization

• All Scrapyd API Supported

  • Deploy project, Run Spider, Stop job
  • List projects/versions/spiders/running_jobs
  • Delete version/project

• Enhancements

  • Basic auth for web UI
  • HTML caching for the Log and Stats page
  • Auto eggify your projects
  • Email notice with multi-triggers provided

Devolopers

• my8100
• simplety (Front-End)

Installation

To install ScrapydWeb, simply use pip:

pip install scrapydweb

Start Up

1. Run scrapydweb -h to get help, then a config file named scrapydweb_settings_vN.py (N for a number) would be copied to current working directory, in which you can custom settings.
2. Run scrapydweb
3. Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome to get better experience.)

Settings

default_settings.py

Screenshots

• Overview

• Dashboard

• Log Analysis

  • stats collection

  • progress visualization

  • logs categorization

• Deploy Project

• Run Spider

• Manage Projects