Socket Disconnect Constantly about phirehose

[DarrenCook]

Are you getting tweets before the socket disconnects, each time? What are some sample run times and tweet counts before each disconnect? Is it disconnecting when the feed is at its quietest, or at its noisiest?

[zachnoren]

I was getting tweets before the socket disconnects pretty consistently but since I have been testing it tonight I have noticed that maybe twice out of every ten reconnects I won't get any tweets at all. It has stayed connected anywhere from about 10 minutes at the most to about 3 minutes at the usual minimum. The frequency is pretty random. I know from past experience that the average statuses/sec or consume rate usually is around 25-27. It does have its anomalies and will peak around 44. If you want some sample output I can get some from console. Also, I am not using any filters.

[DarrenCook]

Based on that, my guess would be your ISP is killing your sockets. A bit of a guess, but reasonably easy to confirm: try setting up on another server. (BTW, you are sure you do not have two instances running, trying to use this account? I imagine that would give the same behaviour you see.)

Assuming you know enough linux, you could set up on an EC2 micro instance quite quickly, easily and cheaply; if that also doesn't work it suggests the problem is your account, or your script. If it does work, it suggests the problem is your ISP or something about OSX PHP. To eliminate the latter, try and find some online Mac cloud hosting?

[zachnoren]

Ugh, thanks Comcast. There was a reason I got a business class plan and it was so I could do this stuff. Tried on another server (Ubuntu 14) and the issue goes away. Also made sure I didn't have two instances running at the same time. The problem with a remote server is the storage costs are outrageous. I have enough home storage to be able to store a year's worth of data and analysis. I also can't transfer data as fast as it is coming in from a remote to my home server. I guess I don't really have any other options. I appreciate all the help and thanks for all the contributions.