coolster1 / dark-rt-toolkit

Toolkit for detecting dark retweets off of the random Twitter Streaming API

dark-rt-toolkit

Toolkit for detecting dark retweets off of the random Twitter Streaming API. This toolkit was used as part of the PhD thesis on "Dark Retweets: Investigating Non-conventional Retweeting Mechanisms" by Norhidayah Azman, University of Southampton, 2014.

Pre-requisite packages

The following packages must be installed in order to make the scripts work:

• Python 2.7
• Python package: numpy
• Python package: tweepy
• Python package: longurl
• Python package: simplejson

Toolkits 1 and 2

dark-rt-toolkit/tk1-random is for retrieving a random sample, where tweets are selected at random from the Twitter stream, meaning that the number of sampled tweets is close to the number of sampled URLs.

dark-rt-toolkit/tk2-url-drill-down is for retrieving URL drill-downs, where all the tweets which contain a given URL gets classified by the same toolkit, meaning that the number of sampled tweets is much higher than the number of sampled URLs, as each URL is represented by many tweets.

User inputs

a_main_recovery.py

• prefix:

  • Change to your chosen path
  • This folder will store all the CSV files generated by this toolkit

• user_limit

  • Change to your chosen size
  • This integer will determine the number of times the Streaming API will be pinged to request random tweets

ratelimit.py

Up to four (4) different OAuth sets of keys are expected to run this toolkit. This is to swap keys when the rate limit for one set of keys have been exhausted. These keys are hard-coded into the code.

• getKeyData()
  • change 'consumer_key0', 'consumer_secret0', 'key0', 'secret0' etc to all four sets of OAuth keys respectively.

If more (or less) than four keys are being included, then edit if-then-else statements in getKeys(), getKeyData() and waitKey() accordingly.

count_rough_blocks.py

• prefix:
  • Change to your chosen path
  • This folder will store all the CSV files generated by this toolkit

reach_counter_tweets.py

• prefix:
  • Change to your chosen path
  • This folder will store all the CSV files generated by this toolkit

How to use

1. Edit the user inputs as described above to suit your requirements.

2. Run a_main_recovery.py

3. If program hangs whilst running, quit program and re-run a_main_recovery.py until completed.

4. Run count_rough_blocks.py to get the aggregated count of all dark and visible retweet types.

5. Run reach_counter_tweets.py to get the reach of all dark and visible retweet types.