vspiewak / twitter-sentiment-analysis

Streaming tweets with spark, language detection & sentiment analysis, dashboard with Kibana

Twitter Sentiment Analysis

• retrieve tweets using Spark Streaming
• language detection
• sentiment analysis (StanfordNLP)
• index tweets in Elasticsearch
• live dashboard using Kibana

Docker setup

Configure Twitter OAuth in docker-compose.yml file.

After installing docker & docker-compose, you can launch the stack with the following commands:

docker-compose build
docker-compose up

After a while, Elasticsearch, Kibana, and Spark should be up:

• http://0.0.0.0:4040/streaming
• http://0.0.0.0:9200/_plugin/head
• http://0.0.0.0:5601

Manual setup

curl -O http://d3kbcqa49mib13.cloudfront.net/spark-1.5.2.tgz
tar xvzf spark-1.5.2.tgz
cd spark-1.5.2
./sbt/sbt assembly

cd ..

curl -O https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.1/elasticsearch-2.1.1.tar.gz
tar xvzf elasticsearch-2.1.1.tar.gz
cd elasticsearch-2.1.1
bin/plugin -install mobz/elasticsearch-head
bin/elasticsearch -d

cd ..
chmod a+x insert.dashboard.sh
./insert.dashboard.sh

curl -O https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz
tar xvzf kibana-4.3.1-linux-x64.tar.gz 
cd kibana-4.3.1-linux-x64
bin/kibana

cd ..

curl -L -O https://dl.bintray.com/sbt/native-packages/sbt/0.13.9/sbt-0.13.9.tgz
tar xvzf sbt-0.13.9.tgz

JAVA_OPTS=-Xmx2G sbt/bin/sbt assembly

../spark-1.5.2/bin/spark-submit \
--class com.github.vspiewak.TwitterSentimentAnalysis \
--master local[2] \
target/scala-2.10/twitter-sentiment-analysis-assembly-0.1-SNAPSHOT.jar \
<consumer_key> \
<consumer_secret> \
<access_token> \
<access_token_secret> \
[<filters>]

twitter-sentiment-analysis's Issues

Hi

I am really interested to know about your project as I found it very interesting. If possible, could you please explain me how your app works?

Exception in thread "main" java.lang.NoClassDefFoundError: com/cybozu/labs/langdetect/DetectorFactory

Hi,

I was trying your example and I see this error "Exception in thread "main" java.lang.NoClassDefFoundError: com/cybozu/labs/langdetect/DetectorFactory". Could you please suggest what I am missing? I added the line libraryDependencies += "com.cybozu.labs" % "langdetect" % "1.1-20120112" to build.sbt too.

Docker - Spark installation problem

Hello,

I download last version if git and I have this issue :
Can you help me ?

docker-compose build
Building elasticsearch
Step 1 : FROM elasticsearch:2.1.1
---> 6f609da577b7
Step 2 : MAINTAINER vspiewak
---> Using cache
---> c2287a0451fa
Step 3 : ENV PATH /usr/share/elasticsearch/bin:$PATH
---> Using cache
---> b90d29d86c3a
Step 4 : RUN plugin install mobz/elasticsearch-head
---> Using cache
---> 2e45724c17af
Successfully built 2e45724c17af
Building spark
Step 1 : FROM java:8
---> 6c9a006fd38a
Step 2 : MAINTAINER vspiewak
---> Using cache
---> 1cb8abf01c10
Step 3 : WORKDIR /opt
---> Using cache
---> 5a1a9f990453
Step 4 : ENV SPARK_VERSION 1.5.2
---> Using cache
---> c2b58646147e
Step 5 : RUN curl -o spark.tgz http://d3kbcqa49mib13.cloudfront.net/spark-$SPARK_VERSION.tgz
---> Using cache
---> cdcc4a3cb188
Step 6 : RUN tar xvzf spark.tgz
---> Using cache
---> 10f720c58dc4
Step 7 : RUN rm spark.tgz
---> Using cache
---> ea879762a036
Step 8 : RUN ln -s spark-$SPARK_VERSION spark
---> Using cache
---> 3ae136d6e56e
Step 9 : WORKDIR spark
---> Using cache
---> 87283edd276f
Step 10 : RUN sbt/sbt assembly
---> Running in 60e6788f38de
NOTE: The sbt/sbt script has been relocated to build/sbt.
Please update references to point to the new location.

  Invoking 'build/sbt assembly' now ...

Using /usr/lib/jvm/java-8-openjdk-amd64 as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
Attempting to fetch sbt
Our attempt to download sbt locally to build/sbt-launch-0.13.7.jar failed. Please install sbt manually from http://www.scala-sbt.org/
ERROR: Service 'spark' failed to build: The command '/bin/sh -c sbt/sbt assembly' returned a non-zero code: 255

Loads of things replaced by stubs

Sorry to be such a noob, can the build recover from things like the following?

info] Compiling 17 Java sources to /opt/spark-1.5.2/unsafe/target/scala-2.10/classes...
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * com.google.guava:guava:(12.0.1, 11.0.2) -> 14.0.1
[warn] Run 'evicted' to see detailed eviction warnings
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * com.google.guava:guava:11.0.2 -> 14.0.1
[warn] Run 'evicted' to see detailed eviction warnings
[info] Compiling 6 Scala sources and 3 Java sources to /opt/spark-1.5.2/external/flume-sink/target/scala-2.10/classes...
[warn] Class com.google.common.collect.ImmutableMap not found - continuing with a stub.
[warn] Class com.google.common.collect.ImmutableMap not found - continuing with a stub.
[warn] Class com.google.common.collect.ImmutableMap not found - continuing with a stub.
[warn] Class com.google.common.collect.ImmutableMap not found - continuing with a stub.
[warn] four warnings found

execution of insert.dashboard.sh script fails

Hi, reporting an issue here, when i try to run the script, it returns me this:

MacBook-Pro-Retina-de-Alonso:twitter-sentiment-analysis aironman$ sh insert.dashboard.sh {"took":4,"errors":true,"items":[{"index":{"_index":".kibana","_type":"config","_id":"4.0.0","_version":4,"_shards":{"total":2,"successful":1,"failed":0},"status":200}},{"index":{"_index":".kibana","_type":"index-pattern","_id":"twitter","_version":4,"_shards":{"total":2,"successful":1,"failed":0},"status":200}},{"index":{"_index":".kibana","_type":"search","_id":"All-tweets","_version":4,"_shards":{"total":2,"successful":1,"failed":0},"status":200}},{"index":{"_index":".kibana","_type":"visualization","_id":"Tweets-count","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"json_parse_exception","reason":"Unexpected close marker '}': expected ']' (for root starting at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@4e5344b; line: 1, column: 0])\n at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@4e5344b; line: 1, column: 309]"}}}},{"index":{"_index":".kibana","_type":"dashboard","_id":"Overview","_version":4,"_shard

this is the output of elasticsearch:

[2016-10-03 16:32:16,350][DEBUG][action.bulk ] [Amun] [.kibana][0] failed to execute bulk item (index) index {[.kibana][visualization][Tweets-count], source[{"title":"Tweets count","visState":"{\"type\":\"metric\",\"params\":{\"fontSize\":60},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}}],\"listeners\":{}}","description":"","savedSearchId":"All-tweets","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"filter\":[]}"}}}]} MapperParsingException[failed to parse]; nested: JsonParseException[Unexpected close marker '}': expected ']' (for root starting at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@33d22f76; line: 1, column: 0]) at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@33d22f76; line: 1, column: 309]]; at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:156) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309) at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:580) at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:559) at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:211) at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:648) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected close marker '}': expected ']' (for root starting at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@33d22f76; line: 1, column: 0]) at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@33d22f76; line: 1, column: 309] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) at com.fasterxml.jackson.core.base.ParserBase._reportMismatchedEndMarker(ParserBase.java:559) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:752) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53) at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:139) ... 18 more

Running on osx Sierra.

Another little issue is that the provided versions of kibana and elastic search does not work, i had to install them using the brew command.