scraping-service's Introduction

Scraping Service

Scraping Service is a REST API for scraping dynamic websites using Node.js, headless Chrome and Cheerio.

Made by the team at Weld (www.weld.io), the #codefree web/app creation tool:

How to Run

Start Chrome in headless mode first:

# Mac OS X
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --headless --disable-gpu --remote-debugging-port=9222

Then start Scraping Service with:

npm run-script dev # development

npm start # production

Server will default to http://localhost:3036

How to Test

npm test

How to Use

Do a HTTP GET:

http://localhost:3036/api/scrape?url=https://news.ycombinator.com&selector=.title+a

Results:

{"time":792,"results":[{"selector":".title a","count":61,"items":["Ask a Female Engineer: Thoughts on the Google Memo", ...]}]}

Parameters:

url (required)
selector is a JQuery style selector, defaults to body. You can use multiple selectors separated by comma, which leads to more items in the results array.
time e.g. &time=5000 adds extra loading time before accessing DOM.
deep set to true to get recursive object trees, not just first-level text contents.
complete set to true to get full DOM nodes, not just text contents.

Implementation

Built on Node.js, Express, headless Chrome, Cheerio.

Deploying on Heroku

Stack: Heroku-16

Buildpacks:

1. https://github.com/heroku/heroku-buildpack-google-chrome
2. heroku/nodejs

Procfile:

web: /app/.apt/usr/bin/google-chrome & node app/server.js

Heroku set-up

# Set up and configure app
heroku create MYAPPNAME
heroku config:set NODE_ENV=production

# Stack and Buildpacks
heroku stack:set heroku-16
heroku buildpacks:add --index 1 https://github.com/heroku/heroku-buildpack-google-chrome

Recommend Projects