log info about dropped tweets about twarc

[remagio]

Current release seems much better than previous releases. I'm now archiving live a streams of a special events, for sure it will be the biggest event of the country.
Just curious about results and statistics to share.

[remagio]

I monitored live an event hashtags with nearly reaching 97.464 TW with ~400k RT (~3hour event).
Ridiculous, but huge numbers here. I got only 238 "No JSON object could be decoded".
imho My opinion is that it's not really all related with timeout/load: I was close to the "firehouse", at favorite time and most were not lost during load interactions.

[edsu]

That's great to hear! I still think it would be good to log any information that Twitter provides about tweets it couldn't fit down the wire. I've heard they send it I just need to find out where it is.

[remagio]

Now the bad news, I'm thinking this getting issue #59.
I ran many --stream and if it takes only a few hours it gets only a few dropped connection. Past weeks I twarc_ed_ only events for 3 hours and all run smoothly.
But using it over 24h it became a mess. Same stopping and restarting it. I noticed that if I restart it waiting more hours, next morning, it runs smoothly again.
Are Twitter testing new kind of API limits?

[edsu]

A dropped connection doesn't necessarily mean dropped tweets. It just means the connection between you and twitter was disrupted. It should re-establish a connection?

[remagio]

Yes, it reestablish connections.
I added previous comments because doing many hydrate (issue #59) my idea is that it's not kind disrupted connection but kinda new moderations. Not disruptions.
How can it be disrupted at every iteration at same 5 ids for 3 times in a raw, waiting too 2h each before iteration ?
imho best time spent in getting a clear error msg than tuning twarc for this.

[edsu]

@remagio this issue is more about how to capture information that Twitter provides about dropped tweets during high volume of activity on the filter stream.