xiya233 / stilio

🦎 A DHT crawler with a minimalistic frontend, written in Python

stilio

Goals

• Completely written in python.
• Fast. But *never* trade code readability for speed.
• Easy set up.

Quickstart

A docker-compose file is included to allow a fast and easy set up.

docker-compose up

Note that a it's recommended to run the frontend and database separated from the crawler since the last one generates a lot of traffic and can make a poor experience for the users. You can read a guide about how to do this here.

FAQ

The crawler speed is not so good.

Try to run this commands if you are on linux:

iptables -I OUTPUT -t raw -p udp --sport PORT_NUMBER -j NOTRACK
iptables -I PREROUTING -t raw -p udp --dport PORT_NUMBER -j NOTRACK