doudou20188 / hot-crawler

A springboot-based hot information crawler. 一个基于 Spring Boot 的热点信息爬虫。支持多线程、高可扩展性。

hot-crawler

Languages: English | 中文简体

What's hot-crawler

hot-crawler is a web crawler about hot information on excellent websites. It can help you quickly access valuable information on many websites. It is compatible with multiple devices, such as personal computers, mobile phones, and tablets. These websites are mainly composed of two languages, Chinese and English. As shown in the figure below.

How to develop

Before starting

To develop Hot-Crawler, you need to install the following softwares:

• Git
• JDK
• Maven
• Redis

We strongly suggest using IntelliJ IDEA to develop this project.

How to run

How to run with Maven command line

1. Clone project source to your local computer

   $ git clone https://github.com/tagnja/hot-crawler.git
   

2. Active develop configuration in src/main/resources/application.yml

   spring:
     profiles:
       active: dev
   

3. Startup your local Redis Server before running the project.

4. Running the project with Spring Boot maven plugin.

   cd hot-crawler
   mvn spring-boot:run
   

5. Visting the website by http://localhost:8080

Document

For example of crawler development to view crawler development tutorial.

For using techniques of the project to view use techniques list.

For the complete crawled websites to view websites crawler list.

For more info about update log to view update log.

Contributing

Contributing is welcome! Please check out the Contributing to hot-crawler guide.

License

hot-crawler is released under the MIT License.