sentdex / socialsentiment

Sentiment Analysis application created with Python and Dash, hosted at socialsentiment.net

Social Sentiment Dash Application

Live-streaming sentiment analysis application created with Python and Dash, hosted at SocialSentiment.net.

Dash Tutorials

This application was created in conjunction with the Dash tutorial series.

Repo Contents:

• dash_mess.py - This is currently the main front-end application code. Contains the dash application layouts, logic for graphs, interfaces with the database...etc. Name is descriptive of the overall state of code :) ...this code is setup to run on a flask instance. If you want to clone this and run it locally, you will be using the dev_server.py
• dev_server.py - If you wish to run this application locally, on the dev server, run via this instead.
• twitter_stream.py - This should run in the background of your application. This is what streams tweets from Twitter, storing them into the sqlite database, which is what the dash_mess.py file interfaces with.
• config.py - Meant for many configurations, but right now it just contains stop words. Words we don't intend to ever count in the "trending"
• cache.py - For caching purposes in effort to get things to run faster.
• db-truncate.py - A script to truncate the infinitely-growing sqlite database. You will get about 3.5 millionish tweets per day, depending on how fast you can process. You can keep these, but, as the database grows, search times will dramatically suffer.

Quick start

• Clone repo
• install requirements.txt using pip install -r requirements.txt
• Fill in your Twitter App credentials to twitter_stream.py. Go to apps.twitter.com to set that up if you need to.
• Run twitter_stream.py to build database
• If you're using this locally, you can run the application with the dev_server.py script. If you want to deploy this to a webserver, see my deploying Dash application tutorial
• You might need the latest version of sqlite.

sudo add-apt-repository ppa:jonathonf/backports
sudo apt-get update && sudo apt-get install sqlite3

• Consider running the db-truncate.py from time to time (or via a cronjob), to keep the database reasonably sized. In its current state, the database really doesn't need to store more than 2-3 days of data most likely.

Tips for Running on Server

• You can use Gunicorn to run the server

gunicorn dash_mess:server -b 0.0.0.0:80 -w 4

Todo

Want to help contribute???

• Code is ugly. Low hanging fruit is just making the code not so ugly. Up to this point, I've just been in "make it work" mode.
• App is ugly. I am not a designer. This app is prettttttyyyyyy gross. Think you have a better design? Halp.
• Click-able related terms and trending terms would be nice. I tried, but failed at this. It'd be cool to see a related term, and be able to just click on it, and this becomes the new searched term, for example.
• The interactive search is cool, but also does a search in the database per-character. It would be nice if it didn't search per key-press. Not sure I want a search button, I like the streamlined interactivity, but maybe wait 0.2 seconds or something without any new keypresses to perform the search? Something like that might help with speeds. I really do not know the best option here, I just know this isn't idea.
• Other manipulations or ideas for interactivity? Feel free to show them in a PR.

Credits

The speed of the application, especially with a database with 10's of millions of records is thanks entirely to Daniel Kukiela who helped us to convert from regular sqlite to using fts, helping with the queries, new database structure, caching, and more.

socialsentiment's Issues

Broken socialsentiment app

Hi Harrison,
Would you be able to activate/fix the socialsentiment.net app that you have in the Readme file. The connection to the Twitter API doesn't see to be working anymore.

We highlight your app in our new Dashboard apps webpage (under the "Connecting to API" category) that gets thousands of views daily.

Thank you,
Adam Schroeder
Plotly Community Manager

https://cdnjs.cloudflare.com/ajax/libs/materialize/0.100.2/js/materialize.min.js is not available

please how solution https://cdnjs.cloudflare.com/ajax/libs/materialize/0.100.2/js/materialize.min.js is not available

In twitter_stream.py

401
401
401
401
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 000007FED0BE94C4 Unknown Unknown Unknown
kernel32.dll 0000000077774B93 Unknown Unknown Unknown
kernel32.dll 00000000777359ED Unknown Unknown Unknown
ntdll.dll 000000007786B371 Unknown Unknown

Why not use redis for caching?

I saw the code is using a lot of sqlite for caching, might if I asked why not redis?

sqlite3.ProgrammingError: Recursive use of cursors not allowed.

While searching for new term this app throws the following error.

  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/dash/dash.py", line 967, in dispatch
    return self.callback_map[target_id]['callback'](*args)
  File "/home/n3011/miniconda3/envs/twitter/lib/python3.6/site-packages/dash/dash.py", line 907, in add_context
    output_value = func(*args, **kwargs)
  File "/home/n3011/work/twitter/socialsentiment/dash_mess.py", line 228, in update_pie_chart
    sentiment_pie_dict = cache.get('sentiment_shares', sentiment_term)
  File "/home/n3011/work/twitter/socialsentiment/cache.py", line 41, in get
    result = self.cursor.execute("SELECT value FROM {} WHERE key = ?".format(pool), (key,)).fetchone()
sqlite3.ProgrammingError: Recursive use of cursors not allowed.

cannot import name 'Event' from 'dash.dependencies'

Looks like the Event system got removed in the latest version of Dash. Is there a reference to know what versions of packages were installed at the time this tutorial was being created?

Error with sentiment_fts table

I found a error during running twitter_stream.py and dev_server.py like this:
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT sentiment.* FROM sentiment_fts fts LEFT JOIN sentiment ON fts.rowid = sentiment.id WHERE fts.sentiment_fts MATCH ? ORDER BY fts.rowid DESC LIMIT 10': no such table: sentiment_fts

I supposed the cause of this error is in twitter_stream.py when it was creating sentiment_fts table, but I cannot find anything wrong with it. Could anyone please help me with it?
fts5 was already enabled.

Tweet Language

Aren't Vader/TextBlob not accurate for foreign languages?

I believe twitter allows you filter out to just english language tweets so would that be something to pursue?

https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters

Sentiment analysis for tweets written in other language

Hi! First of all, thanks for your amazing work!

I would like to share my work based on this great repository, I made sentiment analysis dashboards from Portuguese language, check it out! You are able to change the language.

• https://github.com/Alro10/twitter-sentiment-live

Cheers.

issue when i run twitter_stream.py "index id_unix already exists"

when i run twitter_stream.py i got this mssg

index id_unix already exists
401
401

no such module: fts5

python twitter_stream.py displays the following message and does not continue.

no such module: fts5

Hosted example site failing

FYI: The SocialSentiment.net link, which I'm assuming is the example site hosted on a live server, seems to be failing. A Error loading dependencies error is being raised.

Browser: Version 64.0.3282.119 (Official Build) (32-bit)

Missing requirement and spelling-error

The master-source as of 17.09.2019 is missing a requirement and has a grammar-error.

Requirements.txt is missing "textblob", to install it over pip simply type
pip install textblob
or click here get informations how to install it or simply open requirements.txt and simply add
textblob
into a new line.

Under twitter_stream.py, in Line 32 is a " missing to close out the variable asecret. That isnt a big deal thought, youre supposed to enter your asecret manually in the source-code in there so you most likely will notice that yourself.

Im new on github, and idk how to create pull-requests. :(

whitespace character support

Do we need some fix for searching tweet exclusively on basis of whitespace character?
For example, I need to search sentiments related to the word "intern", not internet nor international.
So we could add support for '\s' too.

Keep getting the error message: no such table: sentiment_fts

pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT sentiment.* FROM sentiment_fts fts LEFT JOIN sentiment ON fts.rowid = sentiment.id WHERE fts.sentiment_fts MATCH ? ORDER BY fts.rowid DESC LIMIT 10': no such table: sentiment_fts

Any Solution?

OS: MacOS high sierra