The thresher from pombredanne

thresher's Introduction

Thresher

Thresher is a library for modern web scraping in Node.js. It is unique in that:

it is headless: URLs are rendered in a GUI-less browser, meaning the version of the HTML you scrape is the same one visitors would see on their screen
it is declarative: Scrapers are defined in separate JSON files. This means no programming required! It also means any other software supporting the same format could use the same scraper definitions.

Thresher was developed as part of the ContentMine stack for mining the academic literature.

Installation

npm install --save thresher

Usage

See quickscrape for a thorough example of thresher in action.

Contributing

We are not yet accepting contributions, if you'd like to help please drop me an email ([email protected]) and I'll let you know when we're ready for that.

Release History

0.0.1 - fork thresher from quickscrape
0.0.2 - solve the dependency problem by exposing binaries from installed packages
0.0.3 - fix rendering issue on some websites
0.0.4 - automatic scraper selection
0.0.5 - fix bug in ScraperBox for multiple matches
0.0.6 - refactor, headless rendering optional, multiple URLs, automatic scraper selection
0.0.7 - support scraperJSON v0.0.2: add regex capability, remove all logging, move to event-based design
0.0.8 - resolve URLs with head-follow requests; refactor to have a Scraper class; supply cookies to downloader; use jsdom-lite to forestall some of the worst installation issues (thanks to @darobin).
0.0.9 - fix broken release
0.1.0 - complete rewrite to support scraperJSON v0.1