edwardt / sitecrawler Goto Github PK
View Code? Open in Web Editor NEWThis project forked from forcedotcom/sitecrawler
This is a Java library which can be used to crawl the content of some of web properties (www.salesforce.com, blogs.salesforce.com for example). It supports dynamic scaling (depending on available machine power (CPU, RAM) and network capacity) out of the box. It also has a Plugin structure, which allows others to write code (plugins) that act on the crawled pages.
License: BSD 3-Clause "New" or "Revised" License