A program that crawls text data from a website at any depth level
-
Download the executable file from releases
-
Run:
./web-crawler <website> <depth> <output file>
Example:
./web-crawler github.com 2 output.txt
Output is a text file which contains all the words from the crawled website with their number of appearances:
I 49595
love 38230
you 26928
three 23111
thousand 22471