Errors during Post to createEphemeral: PROTOCOL_ERROR about cloud-sql-proxy

[Carrotman42]

We have confirmed that the issues are due to a bug found in earlier versions of the Cloud SQL Proxy (we believe it's related to googleapis/google-api-go-client#161), triggered by a rollout being done for internal Google infrastructure.

All users should update to the latest version (1.09) to ensure that this problem does not affect you now or in the future.

[thijsterlouw]

This also affected us. It would be good to put this message on the Google Cloud Console dashboard (Status) and/or send an email out, because it took a while before we (via Googling) ended up here...

[josh123a123]

Great way to loose trust Google...

…

On Wed, May 31, 2017, 5:13 PM Thijs Terlouw ***@***.***> wrote: This also affected us. It would be good to put this message on the Google Cloud Console dashboard and/or send an email out, because it took a while before we (via Googling) ended up here... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#92 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADq6i9Y82dDPaBYJQ9hz8UWau30Axv7oks5r_eYSgaJpZM4Nr7RL> .

[Carrotman42]

The underlying infrastructure change which was triggering this bug has been fully rolled back. If you are still encountering this issue please reopen this issue and send an email to [email protected] with the information from the original post.

[Carrotman42]

I apologize for any trust that was lost here; we worked fast to fix the problem and I was not engaged in any decisions about posting publicly until right before @thijsterlouw's post, at which point the rollback was mostly complete.

Please feel free to email me at [email protected] (reference this issue in your mail so that I end up getting forwarded the message) with any specific concerns you have, I'd love to hear you out. If you'd rather converse on GitHub, please feel free to open a new issue where we can discuss what happened there.

[micahnz]

We just experienced this same problem in production but of course just before I got to add GODEBUG it resolved its self. Outage was intermittent and lasted aprox 5 hours.

Cloud SQL Proxy: version 1.11; sha 61590ed built Fri Sep 29 22:26:22 UTC 2017

[ltearno]

Just so that if someone gets the same problem : don't forget to check your network policies. On my pod with the cloud_sql_proxy, I has a network policy installed, which was preventing the cloud proxy from authenticating (need to open the 443 port so it can do OAuth) and connecting to the database (on ports 5432 and 3307 in my context, a postgres database).
So I got those network errors on the proxy side. So to sum up : if you have this problem, first disable your network policies and see if the problem persists. Then adapt your network policies to the cloud_sql_proxy.