Scraping Service is a REST API for scraping dynamic websites using Node.js, headless Chrome and Cheerio.
Made by the team at Weld (www.weld.io), the #codefree web/app creation tool:
Start Chrome in headless mode first:
# Mac OS X
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --headless --disable-gpu --remote-debugging-port=9222
Then start Scraping Service with:
npm run-script dev # development
or
npm start # production
Server will default to http://localhost:3036
npm test
Do a HTTP GET:
http://localhost:3036/api/scrape?url=https://news.ycombinator.com&selector=.title+a
Results:
{"time":792,"results":[{"selector":".title a","count":61,"items":["Ask a Female Engineer: Thoughts on the Google Memo", ...]}]}
Parameters:
url
(required)selector
is a JQuery style selector, defaults tobody
. You can use multiple selectors separated by comma, which leads to more items in theresults
array.time
e.g.&time=5000
adds extra loading time before accessing DOM.deep
set totrue
to get recursive object trees, not just first-level text contents.complete
set totrue
to get full DOM nodes, not just text contents.
Built on Node.js, Express, headless Chrome, Cheerio.
Stack: Heroku-16
Buildpacks:
1. https://github.com/heroku/heroku-buildpack-google-chrome
2. heroku/nodejs
Procfile:
web: /app/.apt/usr/bin/google-chrome & node app/server.js
# Set up and configure app
heroku create MYAPPNAME
heroku config:set NODE_ENV=production
# Stack and Buildpacks
heroku stack:set heroku-16
heroku buildpacks:add --index 1 https://github.com/heroku/heroku-buildpack-google-chrome