guangzhixie / twitter-stream-word-count

Ingest live tweets with Apache Storm and stores word count in Postgres database for analysis

Twitter Stream Word Count

Use Apache Storm to ingest live tweets from Twitter Stream API, and stores word count in Postgres database for further analysis

---

Application Architecture

---

See this document.

---

Steps to run the application

---

1. Create AWS EC2 instance using UCB W205 AMI
2. Make sure all the dependencies are there:
   • Python 2.7
   • virtualenv
   • lein
   • streamparse
   • psycopg2
   • tweepy
   • redis
3. Start Postgres DB
4. Download the project folder to your preferred location
5. Go into the project folder
6. Run dbsetup python script to create databse and table: $ python dbsetup.py
7. Go into tweetwordcount folder: $ cd tweetwordcount
8. Run storm application: $ sparse run
9. You may see the following warning:
   • WARNING: You're currently running as root; probably by accident.
   • Press control-C to abort or Enter to continue as root.
   • Set LEIN_ROOT to disable this warning.
10. Just press enter to continue
11. Application should be running now. You can exit with Ctrl-C

---

Steps to run the serving scripts

---

1. Go into the project folder
2. Go into serves folder $ cd serves
3. Get all the words with their total count of occurrences, sorted alphabetically in an ascending order: $ python finalresults.py
4. Get counts for a particular word: $ python finalresults.py [your word]
5. Get counts for a range ordered by their total number of occurrences: $ python histogram.py [lower],[upper]