catilac / twitter_analytics

How to get it started:

Install packages:

pip install -r requirements.txt

There are 3 migrations which need to be run:

python init_twitter_employees.py # Gets all users into the database
python get_klout_scores.py # Saves the klout scores for the users.
python get_tweets.py # Gets 100k tweets. 

Afterwards, just run python dashboard.py and go to localhost:5000/

This was a fun problem, and now I have a dataset. Unfortunately I don't have enough time to make things perfect, or go for my custom bonus item of doing sentiment analysis.

Luckily, thats something I'm interested in and I'll do that in my free time.

• The graphs are really ugly, and I need to display more detailed information in a window when hovering over a point.

• The top users graph is weird as a scatter plot. The interesting users have a lot of favs or a lot of RTs. There aren't many users with both. Also some users seem to be going into the millions? I would look at some users with less than 10k RTs and Favs. I think there is something there.

• I chose a bad mongodb python lib. I used mongo. pypi said it held a lot of weight, but I should have started with pymongo. The worst part is that for actual web app, I went with pymongo. If I had more time, I'd change that, since it's something that would drive me crazy. It is driving me crazy.

• I used python. I'm really loving it, but there are some things that I need to do research on:

  • Package management: I want to use something like bundler for ruby. I am using X libraries, each with X_i version number and with a command it'll install all of them.

  • Idiomattic python. Could be useful. I'm sure there are nicer ways to do things. I'm not too unhappy because I think the code is readable

twitter_analytics's Issues

Installation

Chirag, what installation procedure should I follow?

If we need to, I can do a skype call later in the week, like friday.

get_tweets.py sometimes get's a 401

Here are some examples of errors.

Progress Report--
        Tweets:  8597 
        Curr User:  KarlConway
Error:  Twitter sent status 401 for URL: 1.1/statuses/user_timeline.json using parameters: (count=200&oauth_consumer_key=uFuIq77xQTgrCXhAj4ugDw&oauth_nonce=15122995949786842743&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1366321829&oauth_token=16999900-Egp9CoOFvp5tGnea7mCAKjxJ77lRiVlrt4vfiRPYZ&oauth_version=1.0&screen_name=Shane_McCauley&oauth_signature=6UO2mDkHtDpzVMzWJH1y8P2wTdo%3D)
details: {"request":"\/1.1\/statuses\/user_timeline.json?count=200&oauth_consumer_key=uFuIq77xQTgrCXhAj4ugDw&oauth_nonce=15122995949786842743&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1366321829&oauth_token=16999900-Egp9CoOFvp5tGnea7mCAKjxJ77lRiVlrt4vfiRPYZ&oauth_version=1.0&screen_name=Shane_McCauley&oauth_signature=6UO2mDkHtDpzVMzWJH1y8P2wTdo%3D","error":"Not authorized"}
Progress Report--
        Tweets:  8789 
        Curr User:  meelymay
Progress Report--
        Tweets:  8989 
        Curr User:  mazdakh
Progress Report--
        Tweets:  9187 
        Curr User:  jsh
Progress Report--
        Tweets:  9387 
        Curr User:  jimprosser
Progress Report--
        Tweets:  9587 
        Curr User:  jeff_currier
Progress Report--
        Tweets:  9787 
        Curr User:  jandet
Error:  Twitter sent status 401 for URL: 1.1/statuses/user_timeline.json using parameters: (count=200&oauth_consumer_key=uFuIq77xQTgrCXhAj4ugDw&oauth_nonce=7597147269282077780&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1366321867&oauth_token=16999900-Egp9CoOFvp5tGnea7mCAKjxJ77lRiVlrt4vfiRPYZ&oauth_version=1.0&screen_name=diepvule&oauth_signature=i1CFtEBj0AjM6UMpZSWHLe04DOQ%3D)
details: {"request":"\/1.1\/statuses\/user_timeline.json?count=200&oauth_consumer_key=uFuIq77xQTgrCXhAj4ugDw&oauth_nonce=7597147269282077780&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1366321867&oauth_token=16999900-Egp9CoOFvp5tGnea7mCAKjxJ77lRiVlrt4vfiRPYZ&oauth_version=1.0&screen_name=diepvule&oauth_signature=i1CFtEBj0AjM6UMpZSWHLe04DOQ%3D","error":"Not authorized"}
Progress Report--
        Tweets:  9987 
        Curr User:  Dangreene93
Progress Report--

Are there really tweets with 1.1M retweets?

The graphs produces give a broad understanding the type of tweets and their engagement.

It would be nice to have additional graphs that help identify Top Users, and Top Content

• Which tweets were most successful?
• Which users get the most engagement?
• Of users that cross an engagement threshold what did the activity look like over the course of a week or more?
• What other good questions?

Do something interesting with the data.

Do something interesting with the data.

it would be nice if 'dashboard' could connect to a remote data set.

it would be nice if 'dashboard' could connect to a remote data set. i imagine the get_tweets.py takes awhile, it would be great if the dashboard could connect to a remote server to render the data graphs.

maybe a separate file with different defaults would be the KISS solution.

How long does it take to run?

How long does it take to run? get_tweets.py

get_klout_scores.py causes a lot of 'ERROR: HTTP Error 403: Forbidden'

I suspect that the API is being rate limited, and I am not sure I am getting full data from Klout.

I ran it twice and got two different set's of responses. Below are the responses.