Comments (4)
Hmm that's strange.
Have you tried doing it from some other machine and with a new IP address. I think the limit is related to the IP address (If you are scraping continuously). Try connecting to some other network or maybe something like a VPN/Proxy.
Let me know how it goes.
Edit: Wait you should not be getting a KeyError, I fixed it in this commit - 9ab4846. Try updating the package.
from tweeterpy.
Hello, i am facing a problem with your code when using threads. i tried multiple thread types. tried to put each instance of your Twitter instance in a separated instance but it's not working for me. whith 1 thread it works completely fine but when i am using multi threads even if each instance has it's own proxy rotating every request. it's still throwing 403 forbidden when calling your login function. is there a solution ? here's my code
list_tokens = open('twitter_tokens.txt', 'r', encoding='utf-8').read().splitlines()
profiles_to_scan = open('profiles_to_scan.txt', 'r', encoding='utf-8').read().splitlines()
all_proxies = open('proxies_crack.txt', 'r', encoding='utf-8').read().splitlines()
def getProxy():
return next(proxy_pool)
class Proxy :
def __init__(self,proxy):
self.user = proxy.split(':')[2]
self.password = proxy.split(':')[3]
self.ip = proxy.split(':')[0]
self.port = proxy.split(':')[1]
proxy_pool = itertools.cycle(all_proxies) # Create a cycle iterator from the proxy list
accounts_pool = itertools.cycle(list_tokens)
def get_account():
return next(accounts_pool).strip()
class Profile():
def __init__(self,profile):
self.profile = profile
def get_followers(self):
proxy_object = Proxy(getProxy())
proxy_url = "http://" + proxy_object.user + ":" +proxy_object.password + "@" + proxy_object.ip + ":" + proxy_object.port + "/"
proxies_formatted = {
"http": proxy_url,
"https": proxy_url,
}
config.PROXY = proxies_formatted
twitter = TweeterPy()
twitter.generate_session(auth_token=get_account())
self.profile = self.profile.strip()
has_more = True
cursor = None
while has_more:
try:
response = None
response = twitter.get_friends(self.profile,follower=True, end_cursor=cursor,pagination=False)
with open(self.profile+'.txt', 'a',encoding='utf-8') as save_followers:
for follower in response['data']:
screen_name = follower['content']['itemContent']['user_results']['result']['legacy']['screen_name']
save_followers.write(screen_name+'\n')
has_more = response.get('has_next_page')
api_rate_limits = response.get('api_rate_limit')
limit_exhausted = api_rate_limits.get('rate_limit_exhausted')
if has_more:
cursor = response.get('cursor_endpoint')
## YOUR CUSTOM CODE HERE (DATA HANDLING, REQUEST DELAYS, SESSION SHUFFLING ETC.)
## time.sleep(random.uniform(7,10))
if limit_exhausted:
raise RateLimitError
except Exception as error:
print(error)
twitter.generate_session(auth_token=get_account())
config.UPDATE_API
def create_and_launch_threads(profile):
profile_client = Profile(profile)
profile_client.get_followers()
return
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
# Submit the function to the executor for each number
futures = [executor.submit(create_and_launch_threads, profile) for profile in profiles_to_scan]
# Wait for all futures to complete and get their results
results = [future.result() for future in concurrent.futures.as_completed(futures)]
from tweeterpy.
Hey @ilonabehn2
Could u please attach the logs? Make sure to strip off any sensitive data.
I will take a look at the code in the meantime.
Thanks
from tweeterpy.
Related Issues (20)
- TypeError: 'type' object is not subscriptable (Error when using older version of python. i.e. less than 3.9) HOT 1
- Too Many Requests HOT 1
- API Couldn't be Updated : 'NoneType' object has no attribute 'group'. - Not able to scrape data from the package anymore API Update Error HOT 12
- Is there a way to set the logging level? HOT 1
- Couldn't generate guest token HOT 3
- handle pagination using end cursor issue - Pagination Issue when total number of results specified HOT 2
- tweeterpy 1.0.10 requires charset-normalizer==3.1.0, but you have charset-normalizer 2.1.1 which is incompatible. HOT 1
- API Issues when attempting to use it? HOT 2
- twitter.get_list_tweets gives an error HOT 2
- demjson3.JSONDecodeError: Object literal (dictionary) is not terminated HOT 3
- Building wheel for Brotli (pyproject.toml) did not run successfully. HOT 6
- [ERROR] :: 429 Client Error: Too Many Requests for url。 I get an error when I get too many tweets HOT 1
- Issues with Parallelism - Issue with Error Handling HOT 3
- get_tweet function with_tweet_replies can not return all the replies ? HOT 5
- this script uses py 3.7
- TypeError: Object of type timedelta is not JSON serializable HOT 3
- Unable to clone the async_await branch HOT 1
- get_user_tweets count has no effect HOT 1
- after 50k followers. throwing unrelevant exceptions. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tweeterpy.