skainine / prefetch

Prefetch is a simple web crawler which detects every unique link in a webpage and its sub-pages and archives it.

Prefetch is simple text based web crawler written using Spring Boot to archive entire text of websites into a database.

It uses jsoup and PostgreSQL.

Configuration

Requirements

• Java 8
• PostgreSQL
• Any IDE which supports Java

Steps

1. Import the project into an IDE
2. Install the required Maven Dependencies
3. Install PostgreSQL if not already installed and create a database named prefetch
4. Change the application.properties file as per your PostgreSQL Access Credentials
5. Run the application

You may also use other databases by simply replacing their dependencies in pom.xml file and changing the application.properties file See also

• Using MySQL
• Using MongoDB
• Other Docs