- Use Kafka to save tweets in JSON (basically producer puts in queue and consumer writes to JSON)
- Use spark to read in JSON
- Do some analysis and create charts
- Categorise tweets (user-mentions, links, hastags etc)
- Hashtags found
- Nr. of hashtags and total tweets relation
- Occurrences of hashtags, links in tweets
- Occurences of two hashtags in tweets - a network graph that shows which two hashtags found in the same tweet
#To run $SPARK_HOME/bin/spark-submit sparkjob.py
- Some charts based on data collected on irma (not for a long time though)
- Some charts based on data collected on usopen (did not collect for a lot of time) in the repo