thomasvn / distributedcrawler Goto Github PK
View Code? Open in Web Editor NEWThis project forked from josephkphan/distributedcrawler
Created a Master Slave Architecture for a distributed crawling system that will crawl a given list of reddit pages. Our master uses a custom weighted Round Robin algorithm to distribute URLs to the slave components.