hhy5277 / crawler-cases-demo

HollyJS Moscow

Crawler cases demo

That the project written especially for HollyJS 2017 in Moscow.

npm test

//OR to run specific demo
npm test ./src/puppeteer-adblock-demo

Each demo written as separate applications. So fill free to change and mutate it. To tests your own tho. All demo are under the ./src directory

Demo list

• cheerio-demo - how to use cheerio on server side
• puppeteer-adblock-demo - how to create adblock logic for puppeteer
• puppeteer-cheerio-demo - wrap puppeteer content to cheerio
• puppeteer-demo - the minimal code needs to run puppeteer
• puppeteer-download-demo - how to download file using puppeteer
• puppeteer-linkedin-cookie-demo - how to scrape site using cookie againts the login - steps.
• puppeteer-login-github - common steps to login somewhere and receive some userfull private information.
• puppeteer-optimization-demo - experiments to optimize puppeteer load speed
• request-demo - common logic to to receive static html
• request-puppeteer-together-demo - how to switch requestjs to puppeteer runtime
• save-from-demo - show how to fight with ubnormal behavior sites, without puppeteer using sanbox on serverside.
• scheme-demo - examples to check SEO optimized sites
• seasonvar-demo - example with complex workflow to download video from common website.

So, how to install it

npm install
mv ./src/config.template.js ./src/config.js

The config used to set some private infos. Like github login / password. Site tokens.

---

Copyright (c) 2017 pirateminds.com. Licensed with The MIT License (MIT)