trandoshan-io / crawler

Go process used to crawl websites

crawler

Crawler is a Go written program designed to crawl website

features

• use tor SOCKS proxy to crawl hidden services
• fast, built using valyala/fasthttp (up to 10x faster than net/http)
• extract both absolute and relative URLs
• use scalable messaging protocol (nats)

how it work

• The Crawler process connect to a nats server (specified by env variable NATS_URI) and set-up a subscriber for message with tag todoSubject
• When an URL is received the crawler start crawling
• When crawling is done, the crawler will publish content to nats server with subject contentSubject and found urls with subject doneSubject

crawler's Issues

Load forbidden content type from somewhere

To allow runtime configuration.

• Load them from the database?
• Pass them using environment variable?

Add travis-ci support

Allow clear web crawling

As said as #6 it would be good to add an environment variable (ALLOW_CLEAR_WEB_CRAWLING) that will change the regex used by the crawler to a regex with no .onion restriction so that the crawler will be able to crawl the clear web.

It'll be better if TOR_PROXY is optional

Hi there,
It will great if TOR_PROXY is optional, coz I can use it to index some website inside intranet network. In this scenario, a LOGIN_PORXY with cookies will help index some content that's must be login.

Update go version

Allow configuration of user-agent

Complete readme

Improve crawling regex

• Add support for relative urls
• Make sure it's only .onion urls
• Add unit tests to validate

Unit test website regex