edenzik / bigdata Goto Github PK

Done and works.
Used a .jar from this guy: https://github.com/arifn/Distributed-Indexing-MapReduce

Command to invoke is as follows:

hadoop jar dist-index.jar input_directory output_directory

We can take the code and reengineer it to our purposes.

Output from input/word_count_test.txt on the HDFS:

eden    word_count_test.txt=>4567000 
kahlil  word_count_test.txt=>3456000 
mike    word_count_test.txt=>1234000 
nick    word_count_test.txt=>2345000

All explained in title

This is done: file is in asciiToUnicode.java.

Basically all you need to do is:

URLDecoder.decode(line, "UTF-8");

Where line is your string. Make sure to import URLDecoder and throw UnsupportedEncodingException.

This is so short and trivial we can do it in place in the file.

Learn stemmer, determine how to use it, whether context-sensitive, cached file input for people.txt