weaming / lagoujob

Job data mining repo for lagou.com

Job crawler of Lagou Job

Introduction

This repository holds the code for job data analysis of Lagou. The main functions included are listed as follows:

1. Crawling job data from Lagou, and get the latest information of jobs about Internet.
2. Proxies are collected from XiCiDaiLi.

Prerequisites

pip install -r requirements.lock
docker run -d --name mongo -p 27017:27017 \
    -v $HOME/data/mongo:/data/db \
    mvertes/alpine-mongo

How to Use

1. clone this project from github.
2. Lagou's anti-spider strategy has been upgrade frequently recently. I suggest you run proxy_crawler.py to get IP proxies and execute the code with PhantomJS.
3. run m_lagou_spider.py to crawl job data, it will generate a collection of Excel files in ./data directory.

LICENSE

Apache-2.0