banka2017 / twitter-monitor Goto Github PK
View Code? Open in Web Editor NEWTwitter Crawler Core and some based apps
Home Page: https://tmapi.nest.moe
License: MIT License
Twitter Crawler Core and some based apps
Home Page: https://tmapi.nest.moe
License: MIT License
getTweets withReply为 false的情况下,获取不到cursor 信息,没法分页
universal.json接口无法翻页,尝试max_id和cursor均无法翻页
they killed the url..
请问定向用户采集和关键词搜索采集,目前依旧可以使用游客认证X-Guest-Token,进行数据采集嘛?
是推主的主页时间线,这种应该要鉴权了吧,请问大佬怎么做到这种, 用curl复制能跑通
你好 BANKA, I am pretty sure that your signature builder (https://banka2017.github.io/twitter-monitor/apps/online_tools/oauth_signature_builder.html) generates wrong signatures when using GraphQL links.
import asyncio
import aiohttp
import orjson
async def test() -> None:
"""Inputs used in signature builder.
1:
oauth_consumer_key = "3nVuSoBZnx6U4vzUxf5w"
oauth_consumer_secret = "Bcs59EFbbsdF6Sl9Ng71smgStWEGwXXKSjYvPVt7qys"
oauth_token = "1749847962413617152-wZGYi1AGGPFWpq5mxU767lMKn6WrCD",
oauth_token_secret = "TmgPWdJomJWfXZKKpMkKCFjCDk8LFrTcyaqEtKe4NYNzv"
method = "GET"
url = "https://api.twitter.com/1.1/users/show.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&user_id=783214"
body = ""
timestamp = "1706110770"
nonce = "OWZiNWZiZjY4OTkxNDdkZmIyMWY5ZjU2YTJhODZhZWM"
Generated signature: 'OAuth realm="http://api.twitter.com/", oauth_version="1.0", oauth_token="1749847962413617152-wZGYi1AGGPFWpq5mxU767lMKn6WrCD", oauth_nonce="OWZiNWZiZjY4OTkxNDdkZmIyMWY5ZjU2YTJhODZhZWM", oauth_timestamp="1706110770", oauth_signature="GZ3cEyTVhLp9NtSQYW%2BebDCH%2BaU%3D", oauth_consumer_key="3nVuSoBZnx6U4vzUxf5w", oauth_signature_method="HMAC-SHA1"'
2:
oauth_consumer_key = "3nVuSoBZnx6U4vzUxf5w"
oauth_consumer_secret = "Bcs59EFbbsdF6Sl9Ng71smgStWEGwXXKSjYvPVt7qys"
oauth_token = "1749847962413617152-wZGYi1AGGPFWpq5mxU767lMKn6WrCD",
oauth_token_secret = "TmgPWdJomJWfXZKKpMkKCFjCDk8LFrTcyaqEtKe4NYNzv"
method = "GET"
url = "https://api.twitter.com/graphql/CO4_gU4G_MRREoqfiTh6Hg/UserByRestId?variables=%7B%22withSuperFollowsUserFields%22%3Atrue%2C%22withSafetyModeUserFields%22%3Atrue%2C%22userId%22%3A%22783214%22%7D&features=%7B%22hidden_profile_likes_enabled%22%3Atrue%2C%22hidden_profile_subscriptions_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22responsive_web_twitter_article_notes_tab_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D"
body = ""
timestamp = "1706110880"
nonce = "YzI3YzQxMjBjYzlkNDE0OTliNGMxM2YxNDEzZWEwMDU"
Generated signature: 'OAuth realm="http://api.twitter.com/", oauth_version="1.0", oauth_token="1749847962413617152-wZGYi1AGGPFWpq5mxU767lMKn6WrCD", oauth_nonce="YzI3YzQxMjBjYzlkNDE0OTliNGMxM2YxNDEzZWEwMDU", oauth_timestamp="1706110880", oauth_signature="9miWy0rpZbYibURJTM4zKBasPVc%3D", oauth_consumer_key="3nVuSoBZnx6U4vzUxf5w", oauth_signature_method="HMAC-SHA1"'
"""
test_cases = [
{
"name": "restful",
"url": "https://api.twitter.com/1.1/users/show.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&user_id=783214",
"authorization": 'OAuth realm="http://api.twitter.com/", oauth_version="1.0", oauth_token="1749847962413617152-wZGYi1AGGPFWpq5mxU767lMKn6WrCD", oauth_nonce="OWZiNWZiZjY4OTkxNDdkZmIyMWY5ZjU2YTJhODZhZWM", oauth_timestamp="1706110770", oauth_signature="GZ3cEyTVhLp9NtSQYW%2BebDCH%2BaU%3D", oauth_consumer_key="3nVuSoBZnx6U4vzUxf5w", oauth_signature_method="HMAC-SHA1"'
},
{
"name": "graphql",
"url": "https://api.twitter.com/graphql/CO4_gU4G_MRREoqfiTh6Hg/UserByRestId?variables=%7B%22withSuperFollowsUserFields%22%3Atrue%2C%22withSafetyModeUserFields%22%3Atrue%2C%22userId%22%3A%22783214%22%7D&features=%7B%22hidden_profile_likes_enabled%22%3Atrue%2C%22hidden_profile_subscriptions_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22responsive_web_twitter_article_notes_tab_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D",
"authorization": 'OAuth realm="http://api.twitter.com/", oauth_version="1.0", oauth_token="1749847962413617152-wZGYi1AGGPFWpq5mxU767lMKn6WrCD", oauth_nonce="YzI3YzQxMjBjYzlkNDE0OTliNGMxM2YxNDEzZWEwMDU", oauth_timestamp="1706110880", oauth_signature="9miWy0rpZbYibURJTM4zKBasPVc%3D", oauth_consumer_key="3nVuSoBZnx6U4vzUxf5w", oauth_signature_method="HMAC-SHA1"'
}
]
for case in test_cases:
headers = {
"Authorization": case["authorization"],
"Content-Type": "application/json",
"User-Agent": "TwitterAndroid/10.24.0-release.0 (310240000-r-0) sdk_gphone64_x86_64/12 (Google;sdk_gphone64_x86_64;google;sdk_gphone64_x86_64;0;;1;2013)",
"X-Twitter-API-Version": "11",
"X-Twitter-Client": "TwitterAndroid",
"X-Twitter-Client-Version": "10.24.0-release.0",
"OS-Version": "31",
"System-User-Agent": "Dalvik/2.1.0 (Linux; U; Android 12; sdk_gphone64_x86_64 Build/SE1A.220826.008)",
"X-Twitter-Active-User": "yes",
}
async with aiohttp.ClientSession() as session:
async with session.get(
url=case["url"],
headers=headers
) as response:
resp = await response.content.read(-1)
print(f'[{case["name"]}] Response:\n {orjson.loads(resp)}\n')
try:
print(f'Rate limits:\n{response.headers["x-rate-limit-limit"]}:{response.headers["x-rate-limit-remaining"]}\n')
except KeyError:
print("Failed to get ratelimits.\n")
await asyncio.Future()
if __name__ == "__main__":
asyncio.run(test())
As a a result, response for restful will return expected info in json as expexcted:
[restful] Response:
{'id': 783214, 'id_str': '783214', 'name': 'X', 'screen_name': 'X', 'location': 'everywhere', 'profile_location': None, 'description': "what's happening?!", 'url': 'https://t.co/bGcvaMApJO', 'entities': {'url': {'urls': [{'url': 'https://t.co/bGcvaMApJO', 'expanded_url': 'https://about.x.com/', 'display_url': 'about.x.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 67436194, 'fast_followers_count': 0, 'normal_followers_count': 67436194, 'friends_count': 0, 'listed_count': 88670, 'created_at': 'Tue Feb 20 14:35:54 +0000 2007', 'favourites_count': 5909, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': False, 'statuses_count': 15124, 'media_count': 2413, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'ACDED6', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme18/bg.gif', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme18/bg.gif', 'profile_background_tile': True, 'profile_image_url':
'http://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/783214/1690175171', 'profile_link_color': '1B95E0', 'profile_sidebar_border_color': 'FFFFFF', 'profile_sidebar_fill_color': 'F6F6F6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'pinned_tweet_ids': [], 'pinned_tweet_ids_str': [], 'has_custom_timelines': True, 'can_dm': False, 'can_media_tag': True, 'following': False, 'follow_request_sent': False, 'notifications': False, 'muting': False, 'blocking': False, 'blocked_by': False, 'want_retweets': False, 'advertiser_account_type': 'promotable_user', 'advertiser_account_service_levels': ['dso', 'dso', 'dso', 'analytics', 'analytics', 'media_studio'], 'analytics_type': 'enabled', 'profile_interstitial_type': '', 'business_profile_state': 'none', 'translator_type': 'regular', 'withheld_in_countries': [], 'followed_by': False, 'require_some_consent': False}
Rate limits:
900:895
where GraphQL will return:
[graphql] Response:
{'errors': [{'message': 'Could not authenticate you', 'code': 32}]}
Failed to get ratelimits.
如果我有大量推文ID,我想批量大规模获取这些推文的详情信息,有方式实现吗?我看statuses/lookup.json等接口已经无法使用了
HomeLatestTimeline这个网页的接口,在app端怎么爬取,你能给我一个接口吗...谢谢大哥
Hi, BANKA2017!
I am thankful to your blog post about twitter.
I found error about downloading/reading twitterspace.
now, my script's AudioSpaceId (graphql:audiospace) return error.
Also, twitter-monitor has error now.
test spaceurl : https://twitter.com/i/spaces/1djGXldPqNyGZ
That spaceurl is ended. I know.
Yesterday, I can read information using twitter-monitor.
But. Cannot read that now using twitter-monitor.
<< graphql:audiospace response >>
{"errors":[{"message":"The following features cannot be null: rweb_video_timestamps_enabled","extensions":{"name":"BadRequestError","source":"Client","code":336,"kind":"Validation","tracing":{"trace_id":"censored"}},"code":336,"kind":"Validation","name":"BadRequestError","source":"Client","tracing":{"trace_id":"censored"}}]}
Thank you for read it.
hi all,
My first question is can we put HTTPS proxies in proxy.txt
???
when I run 'get_and_upload_guest_account.mjs' it always finds 0 account .
I always have the same answer:
autonia@wayne:~$ node /home/autonia/opt/banka/twitter-monitor/apps/open_account/scripts/get_and_upload_guest_account.mjs [Wed Mar 27 2024 01:29:09 GMT-0700 (Pacific Daylight Time)]: #GuestToken Update open account [Wed Mar 27 2024 01:29:09 GMT-0700 (Pacific Daylight Time)]: #GuestToken Update guest token #Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F TypeError: Cannot read properties of undefined (reading 'oauth_token') at GuestToken.openAccountInit (file:///home/autonia/opt/banka/twitter-monitor/libs/core/Core.function.mjs:74:90) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async file:///home/autonia/opt/banka/twitter-monitor/apps/open_account/scripts/get_and_upload_guest_account.mjs:33:9 count: 1 open_account: No guest account
I wonder if I missed something in my configuration
in 'Backend (Cloudflare Workers)'
i must edit the file 'backend/wrangler.toml' and replace '<YOUR_ID>' by the ID given by cloudflare after creating the kv space named 'twitter-monitor-workers-kv' like this :
kv_namespaces = [ { binding = "open_accounts", preview_id = "NOT_NECESSARY_TO_CHANGE_THIS_VALUE_KV", id = "20e51c53ad77777ba75a7b6f44cf1f2a" }
the <SECRET_TOKEN>
is a key completely invented by myself, I just have to put the same key in both files, like this
in backend/.dev.vars
SECRET_WORKERS_KEY="7buwum7PFiFumFRQinZBhz5qWDk4qakjo$QoTi5f"
in scripts/get_and_upload_guest_account.mjs
const key = '7buwum7PFiFumFRQinZBhz5qWDk4qa*kjo$QoTi5f'
in scripts/get_and_upload_guest_account.mjs
if the const endpoint=
link replies 'hello world' then everything is good with the configuration on Cloudflare
const endpoint = 'https://worker-winter-bush-156a.oleaster.workers.dev/upload/account'
现在我想实时监控某个twitter关键词还有好方式吗?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.