osome-iu / osometweet Goto Github PK
View Code? Open in Web Editor NEWOSoMe Twitter tools. Including a package like tweepy but for the v2 Twitter api.
License: MIT License
OSoMe Twitter tools. Including a package like tweepy but for the v2 Twitter api.
License: MIT License
Update documentation to reflect #44 .
Something that worked well for pulling user-account data that handles rate-limits as well as time-dependent errors
def connect_to_endpoint(oauth, params):
"""Downloads data from Twitter based on the `oauth` object passed and the
`params` created with the `create_params()` function.
If time-dependent errors (429, 500, 503)
"""
switch = True
while switch:
response = oauth.get("https://api.twitter.com/2/users", params=params)
# Get number of requests left with our tokens
remaining_requests = int(response.headers["x-rate-limit-remaining"])
# If that number is one, we get the reset-time and wait until then, plus 15 seconds.
# This is caught below as well, however, we want to program defensively, if possible.
if remaining_requests == 1:
buffer_wait_time = 15
resume_time = dt.fromtimestamp( int(response.headers["x-rate-limit-reset"]) + buffer_wait_time )
print(f"Waiting on Twitter.\n\tResume Time: {resume_time}")
pause.until(resume_time)
"""To be safe, we check explicitly for these TIME DEPENDENT errors.
That is, these errors can be solved simply by waiting a little while
and pinging Twitter again. So that's what we do."""
if response.status_code != 200:
# Too many requests error
if response.status_code == 429:
buffer_wait_time = 15
resume_time = dt.fromtimestamp( int(response.headers["x-rate-limit-reset"]) + buffer_wait_time )
print(f"Waiting on Twitter.\n\tResume Time: {resume_time}")
pause.until(resume_time)
# Twitter internal server error
elif response.status_code == 500:
# Twitter needs a break, so we wait 30 seconds
resume_time = dt.now().timestamp() + 30
print(f"Waiting on Twitter.\n\tResume Time: {resume_time}")
pause.unit(resume_time)
# Twitter service unavailable error
elif response.status_code == 503:
# Twitter needs a break, so we wait 30 seconds
resume_time = dt.now().timestamp() + 30
print(f"Waiting on Twitter.\n\tResume Time: {resume_time}")
pause.unit(resume_time)
# If we get this far, we've done something wrong and should exit
raise Exception(
"Request returned an error: {} {}".format(
response.status_code, response.text
)
)
# If we get a 200 response, lets exit the function and return the response.json
if response.ok:
return response.json()
@ChrisTorresLugo this would require that I touch both #2 and #1. I don't need to touch #1 however this endpoint is much more effective with the user-based Oauth1 method.
I copied and pasted the below from the documentation that I just created a PR for (#14)...
As you can see in the second method below, if we want to manually control which authorization type we use when making queries, we have to play with the OsomeTweet._use_bearer_token
field (or parameter, or w/e it's called). That being said, you can see in the WARNING (at the bottom of the message) that you can set this to whatever you want and, if you don't set it correctly, it will break the class.
Would it be valuable to create a method which controls this - ensuring the user only passes in True
or False
? I am happy to do this, just wanted to make sure you guys agreed this was needed.
Let me know whenever. Thanks.
As mentioned above, osometweet
defaults to OAuth 2.0 Bearer Token authorization. If you'd like to use OAuth 1.0a authorization you can do that in two ways.
OsomeTweet
class your bearer_token
osometweet
needs your bearer_token
for OAuth 2.0 Bearer Token authorization. Thus, if you do not provide this token, osometweet
will look for your bearer_token
- not find it - and then only use your user context Twitter keys/tokens (i.e. api_key
, api_key_secret
, access_token
, access_token_secret
) from then on. For example, simply initialize the OsomeTweet
class like this...from osometweet.api import OsomeTweet
api_key = "YOUR_TWITTER_API_KEY"
api_key_secret = "YOUR_TWITTER_API_KEY_SECRET"
access_token = "YOUR_TWITTER_ACCESS_TOKEN"
access_token_secret = "YOUR_TWITTER_ACCESS_TOKEN_SECRET"
ot = OsomeTweet(
api_key = api_key,
api_key_secret = api_key_secret,
access_token = access_token,
access_token_secret = access_token_secret
)
osometweet
uses for different methods. You can manually do this by controlling what osometweet
does with OsomeTweet._use_bearer_token
(boolean). For example:from osometweet.api import OsomeTweet
api_key = "YOUR_TWITTER_API_KEY"
api_key_secret = "YOUR_TWITTER_API_KEY_SECRET"
access_token = "YOUR_TWITTER_ACCESS_TOKEN"
access_token_secret = "YOUR_TWITTER_ACCESS_TOKEN_SECRET"
bearer_token = "YOUR_TWITTER_BEARER_TOKEN"
ot = OsomeTweet(
api_key = api_key,
api_key_secret = api_key_secret,
access_token = access_token,
access_token_secret = access_token_secret,
bearer_token = bearer_token
)
# The below line tells osometweet to NOT use the bearer_token even though it has been provided
ot._use_bearer_token = False # <-------
# You can then switch it back with...
ot._use_bearer_token = True
WARNING:
OsomeTweet._use_bearer_token
can only be set to the boolean valuesTrue
orFalse
- any other value will break the class.
So I was playing around with the user_look_ids()
method and realized that our data retrieval methods sort of clash with the set_X_fields()
methods.
For example...
# Get a list of all user_fields
>>> u_fields = o_utils.ObjectFields.return_user_fields()
>>> u_fields
['created_at', 'description', 'entities', 'id', 'location', 'name', 'pinned_tweet_id', 'profile_image_url', 'protected', 'public_metrics', 'url', 'username', 'verified', 'withheld']
# Set these fields
>>> ot.set_user_fields(",".join(u_fields))
# Try and pull user_objects only passing the fields ["created_at","username"]
>>> response = ot.user_lookup_ids(user_ids = ids2find, user_fields = ["created_at","username"])
# This still returns all of user fields that I set before.
>>> response["data"]
[{'created_at': '2013-12-14T04:35:55.000Z', 'url': 'https://t.co/3ZX3TNiZCY', 'id': '2244994945', 'verified': True, 'description': 'The voice of the #TwitterDev team and your official source for updates, news, and events, related to the #TwitterAPI.', 'location': '127.0.0.1', 'username': 'TwitterDev', 'entities': {'url': {'urls': [{'start': 0, 'end': 23, 'url': 'https://t.co/3ZX3TNiZCY', 'expanded_url': 'https://developer.twitter.com/en/community', 'display_url': 'developer.twitter.com/en/community'}]}, 'description': {'hashtags': [{'start': 17, 'end': 28, 'tag': 'TwitterDev'}, {'start': 105, 'end': 116, 'tag': 'TwitterAPI'}]}}, 'public_metrics': {'followers_count': 514823, 'following_count': 2035, 'tweet_count': 3648, 'listed_count': 1682}, 'pinned_tweet_id': '1293593516040269825', 'protected': False, 'profile_image_url': 'https://pbs.twimg.com/profile_images/1283786620521652229/lEODkLTh_normal.jpg', 'name': 'Twitter Dev'}, {'created_at': '2007-05-23T06:01:13.000Z', 'url': 'https://t.co/8IkCzCDr19', 'id': '6253282', 'verified': True, 'description': 'Tweets about changes and service issues. Follow @TwitterDev\xa0for more.', 'username': 'TwitterAPI', 'entities': {'url': {'urls': [{'start': 0, 'end': 23, 'url': 'https://t.co/8IkCzCDr19', 'expanded_url': 'https://developer.twitter.com', 'display_url': 'developer.twitter.com'}]}, 'description': {'mentions': [{'start': 48, 'end': 59, 'username': 'TwitterDev'}]}}, 'public_metrics': {'followers_count': 6056217, 'following_count': 39, 'tweet_count': 3693, 'listed_count': 12267}, 'pinned_tweet_id': '1293595870563381249', 'protected': False, 'profile_image_url': 'https://pbs.twimg.com/profile_images/942858479592554497/BbazLO9L_normal.jpg', 'name': 'Twitter API'}]
So, basically, whenever we are setting or updating the user_fields
they simply add onto one another. I am not sure that this is the best approach. My expectation as a user is for whatever I have passed into a data retrieval method to override any previously set object fields.
Having said the above, I think this highlights the question of whether we want to keep these object field setting-methods (i.e. set_user_fields()
) separate from the actual data call method (i.e. user_lookup_ids()
) or not.
In my opinion, it makes more sense to convert these set_x_fields
methods to internal methods which we can use under the hood to update the parameters. Then, we can imply which fields are retrievable from a specific endpoint based on the available parameters within its respective method by simply including only what is available. For example, place
fields are available if you're pulling tweet timelines, however, they're not available for your pulling user account information. So we would include "place" as a parameter for the tweet timelines endpoint but not for the user lookup methods.
Hopefully this change wouldn't be too much trouble because I think we can just do the following:
What do you guys think?
Please claim if you think you can get to this.
Let's use variable names that are consistent with the newest twitter docs:
Is there a reason to utilize this docstring format...
:param [str, list, tuple] tids
:returns requests.models.Response
:raises Exception, ValueError
vs. this one?
Parameters:
- tids [str, list, tuple] - twitter id numbers
Returns:
- requests.models.Response
Raises:
- Exception, ValueError
The latter is, IMO, much more readable but I've seen the first a lot and I wasn't sure if there are any functional differences?
Replace print
with logger.info
and logger.error
I think it would be good to build out functions for cleaning the data that osometweet
returns.
I was watching one of the Twitter Twitch sessions about doing this and they shared this module from twarc which can be used to partially flatten the data JSON to create a data frame for a csv file. I don't think we can directly take this and work it into our package, however, I thought it could be good to reference.
I'm thinking that this would require it's own module, maybe called wrangle
or data
? Open to suggestions. I'm thinking of functions like:
wrangle.extract()
which takes an input of a raw tweet dict and then "text", "urls", "hashtags", etc.wrangle.flatten()
which takes a single raw tweet dict and returns it in row formHere is the code and error I got:
>>> import osometweet
>>> import os
>>> bearer_token = os.environ.get("TWITTER_BEARER_TOKEN")
>>> all_fields = osometweet.fields.get_all_avail_fields()
>>> expansions = osometweet.UserExpansions()
>>> oauth2 = osometweet.OAuth2(bearer_token=bearer_token)
>>> ot = osometweet.OsomeTweet(oauth2)
>>> ot.user_lookup_ids('12', fields=all_fields, expansions=expansions)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/matthewdeverna/Documents/GitHub/osometweet/osometweet/api.py", line 216, in user_lookup_ids
return self._user_lookup(user_ids, "id", fields=fields, expansions=expansions)
File "/Users/matthewdeverna/Documents/GitHub/osometweet/osometweet/api.py", line 289, in _user_lookup
"Invalid parameter type: `query` must be" "either a list or tuple."
ValueError: Invalid parameter type: `query` must beeither a list or tuple.
In an effort to tighten up the last bit of code I'd like to take the lead on going through everything with pylint
. Right now, we get a pretty bad score almost entirely due to long lines in the docstrings (my bad ๐ - I've made some changes in my editor which should keep this from happening in the future) but there are some other small things that can be addressed as well.
If no one has any objections, I am going to go through all of the code and tighten things up based on pylint
's recommendations. (BTW, please let me know if you do have an issue with this for some reason and/or you recommend something else!)
However, I was wondering... is there a way to build this type of check into our tests pre-merge? If this is a lot of work, I'm not sure it will be worth it but, if it's easy, I think it could be nice to set up. For example, I think there is a way to get GitHub to block merges unless it passes a pylint
check, however, I don't know what is involved to get this done.
Please let me know your thoughts when you can.
To make it easier to use for other people (e.g., other group members), let's put some documentations in the readme file.
OsomeTweet
class, especially about the two different oauth methodsWhat do we think about adding the twitter-api-v2 topic to the repo?
I think this can only be done by an admin. Based on these instructions, the gear icon is not there to allow me to do this - not sure @yangkcatiu if you are an admin?
Using the full archive search we should be able to get users' entire list of tweets. I need to build this script for the superspreaders project anyways so I figured I would just add it to this repo.
The idea here is that the script should take in a file with one user ID per line and output a single file of tweets per user.
Right now the everything
arg (in all methods) will accept random strings instead of a True
or False
boolean object because it evaluates strings as True
.
For example:
ot.user_lookup_ids(list_tweet_ids[:100],โblahโ)
is the same as
ot.user_lookup_ids(list_tweet_ids[:100],everything=True)
since "blah" is a string and is treated as True.
Add a check like the below
if not isinstance(everything,bool):
raise Exception
I have a script that I am writing that uses the temp-mix
branch's full search functionality to pull retweets of a list of users. You can see that script here, however, it's not really necessary for this question.
Basically, this is a long-running script and I'd like to record in my script's log the output of the rate_limit_manager.py
print statements. What is the best way to do this? Do I just need to change these print statements to logger.info()
lines, or is it more involved than that?
I originally suggested adding **kwargs
for all methods in #23 because I thought that there would be too many options. Looking at the endpoints, I actually think it's easier to just include all options explicitly.
Below is a list of endpoint methods and their available options.
tweet_lookup
ids
expansions
media.fields
place.fields
poll.fields
tweet.fields
user.fields
user_lookup_ids
ids
expansions
tweet.fields
user.fields
user_lookup_usernames
usernames
expansions
tweet.fields
user.fields
get_followers
id
expansions
max_results
pagination_token
tweet.fields
user.fields
get_following
id
expansions
max_results
pagination_token
tweet.fields
user.fields
set_user_fields
set_tweets_fields
set_poll_fields
set_place_fields
set_media_fields
According to Twitter's page about consuming streaming data we want to be able to do the below:
Please claim if you think you can get to this.
When bearer token is not correct or expired, _user_lookup() in api.py is not handling 401 status code.
Allows a user ID to follow another user: /2/users/:source_user_id/following
(POST)
Need to figure out whether the filtered streaming rules persist forever, or they reset at some point.
I opened a question on twittercommunity.com asking exactly this but please do chime in if you've found an answer somewhere.
Despite not being present on Twitter's list of HTTP errors, it seems like when streaming rules are created, they return the successful 201 HTTP status code (details).
If the rate_limit_manager
is enabled, the manager will throw an error even though we had no problems.
We should be able to edit the below line...
... to match either of these codes with something like...
if response.status_code not in [200, 201]:
Right now all of the setting methods (set_place_fields
, set_polls_fields
, etc.) all take a string. What do we think about changing this to take a list of strings? We can handle the "stringification" within the method easily with a join. This seems more straight forward and less likely to lead to errors, IMO.
For example, we are assuming that people are going to build the url string correctly, but maybe it's best to do this for them to ensure it's correct?
If you guys are okay with this, I am happy to make these changes and can take care of #12 at the same time.
If you haven't realized yet, I'm a bit anal about documentation ๐
I'd like to include docstrings like the ones I included in the timelines methods (#41) - i.e. include details on the kwargs
(optional arguments) - for each method.
However, they're a bit long, does this bother anyone? I find myself using the docstrings constantly and get frustrated when docstrings have no information so it would personally make me quite happy to include this information - mostly because it is impossible for me to remember everything.
I will take the lead on this if you guys don't mind.
@yangkcatiu you mentioned in the user-lookup
branch README.md
...
Local development
Clone this project, go to the source directory, type the following command to install the package locally:
pip install -e ./
Can you help me understand what this does exactly? The pip documentation says...
... which isn't very helpful.
From other Googling, it seems like maybe this allows me to import the version of the package that is present in the source folder on my local machine. Whereas, if I don't do this, running import osometweet
would give me whatever the latest version of the package was when pushed to PyPi? I just tried this out and it seems like that is at least one difference, is there anything else that this does?
Please claim if you think you can get to this.
Currently, OAuth{1,2}'s make_request uses requests.get() to contact the API. The filtered stream endpoint requires the use of POST to establish the rules that the stream will be filtered on. Other endpoints in the future might require POST or other methods. To get around this issue, I'll be changing make_requests to use requests.requests(method={"GET", "POST",etc}) and will update all the endpoints to pass the corresponding method to make_request.
It's probably time to extract the OAuth part out of the osometweet class.
The user-lookup-ids.py
script currently uses the bearer_token
for authentication. Using the user-context api keys/tokens allows for much more data to be pulled. Make this switch.
Would be great to build the below into the package somewhere...
def get_fib_scores(rt_counts):
"""
Calculate the FIB-index.
INPUT:
- rt_counts (list) : list of retweet count values for all
retweets sent by a single user.
OUTPUT:
- fib_position (int) : super_spreader index where the retweet
count value (of a sorted list of retweets) is greater than or
cd equal to the position in that list.
"""
rt_counts.sort()
# The "[::-1]" below makes this for-loop iterate from the last number instead of starting at zero.
for fib_position in range(1,len(rt_counts)+1)[::-1]:
if rt_counts[-fib_position] >= fib_position:
return fib_position
# If the above criteria is never met, we return the fib_position as zero.
fib_position = 0
return fib_position
For anyone else seeing this, this is equivalent to the h-index algorithm - the above is the adapted version for the superspreaders project.
Here is a link to an out-of-the-box script that can take data directly from Moe's tavern.
utils.py
?)Since it seems like filtered streaming rules persist unless the user removes them (see #85 for an issue seeking to confirm this is true), a simple function that removes all existing filter rules seems like it would be useful.
Something like the below
def remove_all_filter_rules(ot)
current_rules = ot.get_filtered_stream_rule()
print(f'The current filtered stream rules are:\n{current_rules}\n')
# Get all streaming rule ids and then remove them
all_rules = [rule["id"] for rule in current_rules["data"]]
delete_rule = {'delete': {'ids': all_rules}}
response = ot.set_filtered_stream_rule(rules=delete_rule)
print(f"API response from deleting rules:\n{response}\n")
Currently the json
parameter in this line is not used in the function.
To fix it, lines 138 (below) and 140...
osometweet/osometweet/oauth.py
Line 138 in 6ae26c8
should be...
response = self._oauth_1a.get(url, params=payload, stream=stream, json=json)
The OAuth1a
_make_one_request
internal method includes the stream
and json
parameters. (See here)
I have two questions:
json
parameter is not used, which I assume is a bug, however, whether to include it on the return line or delete it as a parameter depends on the next question...stream
and json
?
Currently, osometweet only supports authentication through OAuth 2.0 bearer token. This form of authentication only allows for read-access. Performing any action that requires write-access, such as BotSlayer's send DM functionality, or accessing non-public fields, such as private engagement metrics, will be rejected by the API.
To solve this issue, we have to implement OAuth 1.0a. Ideally, the library will check whether the requested fields are allowed under the instance's authentication prior to calling the API. e.g. raise an error if authenticated using OAuth 2.0 and trying to post a tweet.
Very often it happens that I just want to make a single call to a function to see what it returns, etc.
As a result, I find myself typing out the below preamble before doing anything with the API all the time and it has become my least favorite thing about the package. ๐ผ
# Initialize the OSoMeTweet object
bearer_token = os.environ.get("TWITTER_BEARER_TOKEN")
oauth2 = osometweet.OAuth2(bearer_token=bearer_token)
ot = osometweet.OsomeTweet(oauth2)
A simple solution is to just set up a utility function that does this for us by drawing on the user's environment variables. I am imagining one for both authorization contexts.
I imagine a loading function, as well as an initializing function for each context. Here is a rough example for the App context...
def load_bearer_token(env_key: str = "TWITTER_BEARER_TOKEN") -> str:
"""
Load Twitter Keys from Local Environment.
Parameters:
-----------
- env_key (str) : The name of the environment variable for your Twitter bearer token. (default = "TWITTER_BEARER_TOKEN")
"""
# Set Twitter tokens/keys.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
print("Loading bearer token...")
bearer_token = os.environ.get(env_key, None)
if bearer_token is None:
raise Exception(
f"No environment variable named `{env_key}`! "
"Make sure to set this from your terminal via:\n\n"
f"\t --> '{env_key}'='<your_twitter_bearer_token>' "
)
return bearer_token
def initialize_osometweet(
env_key: str = "TWITTER_BEARER_TOKEN",
manage_rate_limits: bool = True
) -> osometweet.api.OsomeTweet:
"""
Return an authorized osometweet API object
from which we can make API calls.
Parameters:
----------
- env_key (str) : The name of the environment variable for your Twitter bearer token. (default = "TWITTER_BEARER_TOKEN")
"""
print("Initializing osometweet with oauth2a authentication...")
bearer_token = load_bearer_token(env_key)
oauth2 = osometweet.OAuth2(
bearer_token=bearer_token,
manage_rate_limits=manage_rate_limits
)
return osometweet.OsomeTweet(oauth2)
This approach allows users a bunch of freedom like:
initialize_osometweet()
with no input and get an osometweet.api.OsomeTweet
object right awayFor example, if we want to use the tweet_lookup()
method.
You could do something like the following:
resp = self.ot.tweet_lookup(
tids = random_tweet_id,
everything = True
)
And this would be equivalent to including all expansions
, tweet.fields
, place.fields
, poll.fields
, tweet.fields
, and user.fields
.
Another option would be to include and
all
option for each data object, however, since we have thereturn_objectfield()
methods, I think this might be overkill. It's really easy to just callt_fields = o_utils.return_tweet_fields()
and then placet_fields
in theot.tweet_lookup()
call. As a result, I think it's best to build out the first option.
tweet_lookup
user_lookup_ids
user_lookup_usernames
get_followers
get_following
For me, it would be common to fetch all the fields for tweets and users. So maybe we can add an option for this. Seems we can modify utils.ObjectFields
for this.
I'm not sure to do it in the method level or the class level.
Create docstrings for these methods.
user_lookup_ids()
user_lookup_usernames()
The wrangle module was added, we need to add some wiki examples for each. Probably they should get their own page.
flatten_dict
get_dict_paths
get_dict_val
I will probably need to figure this out for the retweet cascade reconstruction part of the superspreaders project so I will have to look into this anyway. If anyone else wants to dig into this, please feel free to assign yourself and go for it because I probably won't get to this for a couple of weeks.
I liked the structural approach that Kaicheng took in his edits to the user lookup methods - having one base method and two user-facing methods (one for IDs and one for usernames) that feed into the base method - so I will try and do the same thing.
Please add any other thoughts you may have.
Please remove the branches that are no longer used at least from the remote. Otherwise, it could soon get really messy.
You can do so by going to https://github.com/osome-iu/osometweet/branches and click the "trash" icon.
Allows a user ID to unfollow another user: 2/users/:source_user_id/following/:target_user_id
(DELETE)
The headers returned by each API call contain the following fields:
- x-rate-limit-limit: the rate limit ceiling for that given endpoint
- x-rate-limit-remaining: the number of requests left for the 15-minute window
- x-rate-limit-reset: the remaining window before the rate limit resets, in UTC epoch seconds
These fields are documented here. Method calls should parse these fields to determine when to sleep the process and for how long. Performing additional requests when the limit has been reached risks account suspension.
API:
fields/expansions:
utils:
Create scripts that can be used to gather data using the available endpoints.
Here are the endpoints available in osometweet
at the moment:
tweet_lookup
user_lookup_ids
user_lookup_usernames
get_tweet_timeline
get_mentions_timeline
get_followers
get_following
search
sampled_stream
filtered_stream
Please feel free to claim one of these scripts if you're interested! ๐
See the examples/
folder for what is available.
Note: The original comment for this issue has been completely edited from it's initial form b/c it was originally opened as a question/discussion about whether we should do this. Now that we've decided to make this standard practice, I wanted to clean it up.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.