Comments (12)
I am seeing this problem after upgrading GKE to 13.9 and cloud-sql-proxy image to 1.15.
Strange thing is if I 'kubectl exec -it -c /bin/sh' and just do "ps" and "kill " the cloud_sql_proxy process, Kubernetes restarts the container and everything starts working. This method has always worked for me though I can't be sure about the timing as it usually takes me several minutes to get to the kill step.
Why would an API not work the first time a container starts and then suddenly work?
Also, I changed the cloud_sql_proxy command to something like: "timeout 4 $cloud_sql_proxy_command && $cloud_sql_proxy_command" where $cloud_sql_proxy_command contains the full syntax with all the CLI flags.
Even with that, when I redeploy all the Pods the method works 60% of the time i.e. for about 60% of the Pods the second cloud_sql_proxy command works. But for about 40% even the second cloud_sql_proxy command fails with the errors OP posted here.
I am just totally confused and frustrated by the unpredictable nature of this cloud_sql_proxy.
from cloud-sql-proxy.
I met problem again. Just re-create new node-pool and delete old node-pool and problem resolved
from cloud-sql-proxy.
Agreed that this doesn't appear to be a cloudsql-proxy-specific issue, and I'm not quite sure how to go about debugging this other than asking you to SSH to your kubernetes nodes and confirm that the network is indeed broken (or at least that HTTP to accounts.google.com isn't working properly).
If SSH to the nodes doesn't work, it's possible a network issue is preventing some packets from flowing as expected (and SSH is probably failing for the same reason HTTP is). If it does work, I suggest doing some sanity curls to see what works and what doesn't:
curl google.com
curl github.com
curl asdf.com # not usually https, can test is port 443 is broken but 80 works (for some reason...)
curl accounts.google.com
from cloud-sql-proxy.
If it does turn out to be the network, moving your cluster to a different zone may fix things.
from cloud-sql-proxy.
Hm, strange, those sanity curls work just fine. I'll look into moving to another cluster and see if that helps.
from cloud-sql-proxy.
The cloudsql connections magically started working again around 23:07 CEST (on the same cluster). Dunno how this could happen, I'm in contact with Google Cloud support about it.
from cloud-sql-proxy.
Any updates here? I suspect there was just some transient network problem but it'd be good to confirm.
from cloud-sql-proxy.
We didn't really get to the bottom of it. The GCP support guy didn't find anything of note in the logs, neither did we. The main suspect is a Docker networking problem. "Some transient network problem" is probably the best description we have right now.
Probably a coincidence, but 5 days later we actually got a very similar issue, but this one seemed to affect more people. This time it was a stream error: stream ID 1; PROTOCOL_ERROR
, see [1], but this issue also disappeared after a while.
We have now bumped the gke-proxy
image to version 1.09 and have upgraded Kubernetes version on our GKE cluster nodes and we've gone 2 days straight without downtime now (hah!)
Anyway, thanks for the suggestions here, I'm fine with closing this issue as I don't think cloudsql-proxy is to blame for this.
from cloud-sql-proxy.
Yep, that issue is just a coincidence. I've posted on that Cloud SQL Discuss post you linked to explaining the issue yesterday (and there's also #92 about it).
Glad to hear things are stable now. I do suspect that the original issue was simply related to docker or the networking in that zone. Please let me know if there's anything I can do.
from cloud-sql-proxy.
just encountered the same (hopefully-transient?) error trying to use cloudsql-proxy kubernetes pod sidecar to connect to a postgres DB hosted in google cloud SQL
edit: I was just working with a small test cluster, so re-creating the cluster in a different zone and region was fine and the problem went away. re-deploying or waiting 1 hour had no effect.
from cloud-sql-proxy.
Met this problem when running cloudsql proxy as sidecar, image name is gcr.io/cloudsql-docker/gce-proxy:1.11
,
Have been seeing the error from time to time for 1 week now.
2018/12/12 04:24:10 couldn't connect to "my db": Post https://www.googleapis.com/sql/v1beta4/projects/myproject/instances/myinstance/createEphemeral?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp: i/o timeout
, it goes away when I retry.
from cloud-sql-proxy.
When using a VPC-native Kubernetes Engine cluster (which I believe is the default now) you no longer need the cloudsql-proxy but instead can just connect to the private IP of your CloudSQL instances, see https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine
from cloud-sql-proxy.
Related Issues (20)
- Do not require strict order in env variables for connections HOT 2
- Add support for a lazy refresh
- pgbouncer + proxy with transaction pooling is slow HOT 23
- Is there any way to connect to MS SQL using domain credentials? HOT 7
- v2/tests: TestSQLServerAuthentication failed HOT 4
- Telemetry doesn't work with non-ADC
- v2/internal/proxy: TestCheckConnections failed HOT 2
- Connecting to cloud-sql using private-ip sometimes fails with a TLS handshake timeout HOT 7
- Availability of a Container Image on Google Artifact Registry HOT 2
- "Cloud SQL IAM service account authentication failed for user ..." intermittent errors when connecting to Postgres HOT 17
- CSQL_PROXY_ADDRESS requires an IP address but doesn't explicitly state so HOT 6
- 30s+ Hang When Using Manual Token Authentication HOT 7
- v2/internal/proxy: TestClientLimitsMaxConnections failed HOT 1
- v2/internal/proxy: TestClientCloseWaitsForActiveConnections failed HOT 1
- v2/internal/proxy: TestClientClosesCleanly failed HOT 1
- v2/internal/proxy: TestClosesWithError failed HOT 2
- v2/internal/proxy: TestClientConnCount failed HOT 2
- v2/internal/proxy: TestRunConnectionCheck failed HOT 2
- Automatic instance discovery isn't supported in v2 Proxy HOT 5
- Brief summary of the proposed feature
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloud-sql-proxy.