This program downloads tweets from Twitter APIs(v2) using an Academic development accounts. All returned information are stored.
There are four folders to store download tweets:
-
raw_tweets
: raw data from Twitter, including five parts of each request:data
,media
,places
,users
, andpolls
. Each part is stored as a individual .CSV file. Strings in these .CSV files are cleaned by removing newlines, tabs, and commas (,
is replaced by;
). Please refer to Twitter API Documentation.file name
:oldest posted time
lastest posted time
tweets_count
_part
. For example: -
line_tweets
: Five parts are merged together, each row in this file contains all columns returned by the API. The result of each request is a single file. There are 100 lines (withcontext_annotation
) or 500 lines (withoutcontext_annotation
) per file.context_annotation
is like a topic assigned to a tweet.About
context_annotation
:Context annotations: Derived from the analysis of a Tweet’s text and will include a domain and entity pairing which can be used to discover Tweets on topics that may have been previously difficult to surface. At present, we’re using a list of 50+ domains to categorize Tweets.
-
chunks_tweets
: Mergedline_tweets
for better managements of Hadoop systems. Each merger .CSV files have about 50,000 - 500,000 tweets. -
cluster_csvs
: Convertedchunks_tweets
. Some fields are expand for our Hadoop systems.
-
Prerequisite:
- Develop account. Put your keys in
tweet_api_keys.txt
(put this file in the same folder as thedownload_tweet_APIv2.py
).
Consumer API Key: copy_your_key_here Consumer API Secret Key: copy_your_key_here Bearer Token: copy_your_key_here Access Token: copy_your_key_here Access Token Secret: copy_your_key_here
- Pandas
- Develop account. Put your keys in
-
Using
download_tweet_APIv2.py
to download tweets.Edit the parameters in the fucntion of
execute_download
, such assaved_path
,query
,start_time
,end_time
, andchunk_size
.To build a query, you need to read Building a query or How to write search queries. Here is a example:
vaccin OR vaccination OR vaccine OR vaccinate place_country:AU
: tweets in Australia contain wordsvaccin
,vaccination
, orvaccinate
.
300 request per 15 minutes; 500 tweets per requets (or 100 tweet with context_annotation
). So the highest speed is 600,000 tweets per hour. However, due to heavy converting computations, the current speed is about 200 request per 15 minutes, which is 400,000 tweets per hour. In my obersvation, the speed is 200,000 tweets per hour only.
If you do not like this repository, twarc is highly recommanded.
- Adaptive sleeping when break the rate limits.
- Code refining.
- Result merging and compressed.