Job crawler of Lagou Job
This repository holds the code for job data analysis of Lagou. The main functions included are listed as follows:
- Crawling job data from Lagou, and get the latest information of jobs about Internet.
- Proxies are collected from XiCiDaiLi.
pip install -r requirements.lock
docker run -d --name mongo -p 27017:27017 \
-v $HOME/data/mongo:/data/db \
mvertes/alpine-mongo
- clone this project from github.
- Lagou's anti-spider strategy has been upgrade frequently recently. I suggest you run proxy_crawler.py to get IP proxies and execute the code with PhantomJS.
- run m_lagou_spider.py to crawl job data, it will generate a collection of Excel files in
./data
directory.