GOAWAY session_timed_out about pigeon

[hpopp]

In the process of fixing on the reconnect-fix branch. FCM likes to close connections after a few minutes or so, which creates an infinite loop of reconnects (and eventually maxes out CPU).

New intended behavior:
FCM workers will gracefully drop the connection on close and only attempt to connect when there are new pushes to send. This will wipe the existing queue and reset stream IDs, but I think these disconnects are happening due to inactivity, so there shouldn't be pending pushes anyway.

@cstar

[asgoel]

shouldn't we just make the default ping period lower then?

[hpopp]

Tried variations down to 5 minutes with no luck. I'm not sure it respects pings, or when it does, GOAWAY, ENHANCE_YOUR_CALM, too_many_pings start showing up in the logs. The connection intervals seem to get shorter with every subsequent reconnect, and right about 45 minutes after the initial connection it's closing and reconnecting so fast it overloads everything.

[asgoel]

Interesting... Dumb question, but I can't find any official documentation on whether FCM supports HTTP/2. Is this just an enhancement you've tried to make, and thus why we are running into so many problems?

[hpopp]

@cstar wrote the initial implementation. The API works fine except for the reconnect issue, which I'm guessing is due to FCM servers not wanting to persist socket connections until the API is public.

I didn't discover it until I started rolling out onto some staging servers this morning. I should have the patch up pretty soon.

[0x6e6562]

@hpopp I'm seeing this sporadically using pigeon 1.1.0 (c0d18bf) and kabadra 0.3.2 (d7afbcfa):

[error] Got GOAWAY, NO_ERROR, Last Stream: 0, Rest: session_timed_out

With various different stream ids.

Should I be looking into the details of this, or is the session time out just business as usual?

[hpopp]

FCM closes the connection after idling for a minute or two. No big deal, it just reopens the next time a push is sent.

I should probably have Kadabra disable logging by default.

[0x6e6562]

Thanks for the heads up - there doesn't appear to be any material impact on message delivery in any case, so we can just treat this as an ignorable log message.