Comments (7)
Current release seems much better than previous releases. I'm now archiving live a streams of a special events, for sure it will be the biggest event of the country.
Just curious about results and statistics to share.
from twarc.
I monitored live an event hashtags with nearly reaching 97.464 TW with ~400k RT (~3hour event).
Ridiculous, but huge numbers here. I got only 238 "No JSON object could be decoded".
imho My opinion is that it's not really all related with timeout/load: I was close to the "firehouse", at favorite time and most were not lost during load interactions.
from twarc.
That's great to hear! I still think it would be good to log any information that Twitter provides about tweets it couldn't fit down the wire. I've heard they send it I just need to find out where it is.
from twarc.
Now the bad news, I'm thinking this getting issue #59.
I ran many --stream and if it takes only a few hours it gets only a few dropped connection. Past weeks I twarc_ed_ only events for 3 hours and all run smoothly.
But using it over 24h it became a mess. Same stopping and restarting it. I noticed that if I restart it waiting more hours, next morning, it runs smoothly again.
Are Twitter testing new kind of API limits?
from twarc.
A dropped connection doesn't necessarily mean dropped tweets. It just means the connection between you and twitter was disrupted. It should re-establish a connection?
from twarc.
Yes, it reestablish connections.
I added previous comments because doing many hydrate (issue #59) my idea is that it's not kind disrupted connection but kinda new moderations. Not disruptions.
How can it be disrupted at every iteration at same 5 ids for 3 times in a raw, waiting too 2h each before iteration ?
imho best time spent in getting a clear error msg than tuning twarc for this.
from twarc.
@remagio this issue is more about how to capture information that Twitter provides about dropped tweets during high volume of activity on the filter stream.
from twarc.
Related Issues (20)
- How to use Academic API to search archives with language filters? HOT 2
- How to extract the author user name of a retweet? HOT 1
- Twarc API V2 function to get conversations HOT 1
- Is it possible to use "next_token"? HOT 3
- Missing `variants` field in `media.fields`
- Add `include_ext_is_blue_verified=True`
- CLI: Allow to differentiate between 404 and connection timeout HOT 4
- Have twarc2 400 error while searching phrases. HOT 1
- CLI Errors fail to show for invalid input file
- CLI Errors fail to show for invalid input file HOT 1
- twarc2 timeline --no-context-annotations not pulling 500 tweets HOT 2
- Adding query parameters to search terms imported via .txt? HOT 1
- Is it possible to convert twarc jsnol files to twarc2 jsonl files? HOT 2
- Impossible to specify No expansions at all in CLI.
- Implement new compliance Streams
- Cannot find user_verified_type in the tweet metadata HOT 18
- twarc2 hydrate HOT 2
- twarc2 hydrate freezes at 20% HOT 3
- Client forbidden HOT 3
- Forbidden error after few tweets hydrated (403) HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from twarc.