rgrig / twitstat

compute some statistics on twitter statuses

• ./fetch_tweets.py
• ./slice.py
• ./rank_users.py
• ./normalize_urls.py
• ./rank_urls.py

twitstat's Issues

talk graph

Dump a graph whose edges A->B means that A wrote a status mentioning @b (or, configurably, starting with @b). Then compute statistics on this graph:

1. clusters (k-means)
2. pagerank

stop getting old statuses

update_statuses.py should stop getting older statuses once it sees that it knows some of those last fetched.

page design

also, figure out how to populate it

get rid of normalize_urls

The reply from Twitter contains expanded_url.

script to remove nonexistent accounts

Remove nonexistent accounts from db/config.js

Options:
(a) visit twitter.co/screen_name and see if account exist (API for equivalent action?)
(b) check the user_id of the old screen_name and update screen_name if there is a newer one

special people

Some people are near a certain location, are influential, but Twitter doesn't return them in the search (e.g. chinezu says he's from Romania, so Twitter doesn't include him in the results for Bucharest). Even though searching for people (as opposed to places) is expensive (in terms of number of 'calls' to Twitter's API) I should have a small list with 'exceptions' and scrape those too.

retweet

At this time retweets (which use the twitter API, as oppesed to the manual ones) are ignored. They should be collected and taken into account for statistics.

auto cluster

choose alpha automatically in cluster.py
also, output hierarchical clusters?

use extended tweets

Currently, I get tweets using the search API v1.1 (https://api.twitter.com/1.1/search/tweets.json). However, it seems to not return extended tweets, which means that everything after the first 140 characters (including mentions and urls) is missing from the API reply.

To explore (I don't know if they'd solve the problem):

• ditch Search, use timeline: https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/overview
• move to API v2

Using timelines means reducing some functionality, but it's functionality I almost never use: most of the time I just want a list of users as the roots.

Moving to v2 may involve more time than I'm willing to spend.

compute word statistics per user

take into account likes as mentions

If user A likes a tweet by user B, add a link A->B in the graph for pagerank.

weight url mentions by pagerank

ignore words within urls