originlive / spider

A web crawler engine that gathers data, sorts it and outputs it to file. The goal is for it to be fully customizable and extensible, with possibilities for scripting the behaviour of the spiders.

SpideR

NOTE: A lot of functionality is still missing. This will be rectified eventually.

A web crawler engine that gathers data, sorts it and outputs it to file.
The goal is for it to be fully customizable and extensible, with possibilities for scripting the behaviour of the spiders. The emphasis is on speed and ease of embeddance.

Requirements:
libcurl, the c++ wrapper curlpp, and gumbo-parser..
See: https://curl.haxx.se/libcurl/
http://www.curlpp.org/
https://github.com/google/gumbo-parser for downloads.
For arch-linux, curl is to be found in the core repos, while libcurlpp and gumbo-git can be found in the AUR.

Installation: Compile it. For linux there is a makefile that should work.

Settings:
Settings are set through the Settings.json file.
The settings currently available are:
textspeed : int - Does nothing atm.
depth : int - Determines how far it should follow links that are found. Default is 1.
debug : bool - Setting this to 1 sets the verbose flag for the connection. But this ends up in the parsing..
type : unchanged|small|firstcapital|fullcapital - Format of the words stored.

Use:
At the moment there are only 3 commands: help, connect and quit.
Connect url - Attempts to connect to the specified site and gather words and url based on the settings set.
Example:

connect www.google.com

Output goes to Output.txt.

spider's Issues

Todo

TODO:

1. • Make program work with depth > 1. Fix parsing of urls
2. • Dump data regularly to file to avoid too large a tree
3. • Incorrect Settings.json should not crash the program, issue an error message.
4. • Add a quit command
5. • Lacking a Settings.json, the user should be given the option to generate a default one.
6. • Add ability to read robots.txt and avoid sites that should not be accessed (patterns like "?id=")
7. • Make robot more polite (make sure not to flood a site, and make it a default but optional setting)
8. • Add ability to deal with ssl or handshakes. (Curl does this)
9. • More customizability, make settings a polymorphic class that has functions to alter logic? (<< this part will be implemented with a scripting language)
10. • Better linux console.
11. • Sort the libraries, they should be downloadable, rather than embedded already.
12. • Make program more into a state machine, polymorphic