CloudSQL Proxy Fails after Short Period of Time (~12 hours or so) about cloud-sql-proxy

[Carrotman42]

Is it possible for you to post the logs for a failing proxy process? If your app is failing to connect to it, there should be some information in the logs about why connections are being denied (especially if connections were once working).

[GLStephen]

@Carrotman42

The logs have a bunch of basic entries like this:

2017/01/12 14:21:38 New connection for "my-project:us-central1:my-db"
2017/01/12 14:21:38 Client closed local connection on 127.0.0.1:3306

Then they have a few of these:

2017/01/12 14:21:46 New connection for "my-project:us-central1:my-db"
2017/01/12 14:21:46 Instance my-project:us-central1:my-db closed connection

I have one group of these:

2017/01/12 14:22:26 New connection for "my-project:us-central1:my-db"
2017/01/12 14:22:26 couldn't connect to "my-project:us-central1:my-db": googleapi: Error 409: The instance or operation is not in an appropriate state to handle the request., invalidState
2017/01/12 14:22:26 New connection for "my-project:us-central1:my-db"
2017/01/12 14:22:26 Thottling refreshCfg(my-project:us-central1:my-db): it was only called 741.653211ms ago

And one of these:

2017/01/12 16:01:26 Error in accept for {"my-project:us-central1:my-db" "tcp" "127.0.0.1:3306"} on 127.0.0.1:3306: accept tcp 127.0.0.1:3306: accept4: too many open files

It's important to note that usage volume at this time was pretty low. Probably 1000s of DB connections per day. We were running a low volume app in a POC style setup.

Unfortunately, my log rotation looks to have been overly aggressive and I think the previous days were removed so I only have this day of info.

Things to note. I have not adjusted the tcp_keepalive_time on these machines as outlined here:
https://cloud.google.com/compute/docs/troubleshooting Is the cloud proxy effected by that? Is it worth adjusting and rechecking?

[Carrotman42]

This error is the reason the process exits:

2017/01/12 16:01:26 Error in accept for {"my-project:us-central1:my-db" "tcp" "127.0.0.1:3306"} on 127.0.0.1:3306: accept tcp 127.0.0.1:3306: accept4: too many open files

This is probably caused by the "invalidState" error that was returned. Do you know what your database was doing during that time period? You should be able to see a list of operations in the Cloud Console. If there's no operation around then, please email the name of your instance to [email protected] with a link to this issue and we can continue investigating from there.

I suspect that your application is overloading the Proxy with retries when the instance is inaccessible due to this invalidState error. It should be temporary, and ideally the client proxy would stay up rather than crash in this case. Do you know how quickly your application retries when the instance is inaccessible? There may be something the client proxy can do to more quickly deny connections so that it doesn't get overloaded, but ideally your application should be written to retry with exponential backoff when it fails to connect.

[Carrotman42]

Do you have any more info available? Is this still affecting you?

[GEverding]

Hey, I'm not sure if this is the same issue but they sound related. I'm using the cloud proxy in gke with a managed Postgres instance and periodically I'll get timeout exceptions in my app.

'''
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
'''
I lost the logs for the cloud proxy but I'll grab them next time I see it.

Is this issue related?

[Carrotman42]

Hey @GEverding sorry I missed your post. If you are still encountering the issue, please grab the logs from the Cloud SQL Proxy and open up a new issue. We can debug there.

@GLStephen, please reopen this issue if this is still affecting you.

[imiskolee]

aslo has this issue. why process is running,but no port listen ?

[smulder]

hope this helps someone... I use node with the cloud_sql_proxy and it was failing exactly as described... Here's how I handle it on my local testing / process setup.

kill is an npm package
child and childRep are defined as a var outside of the promise block

.catch((err)=>{
     if(err.code == 'ECONNREFUSED'){
       var kill = require('tree-kill');
       if(err.port == '3306'){
         if(!child){
           child = spawn('../cloud_sql_proxy -instances=YOUR:INSTANCE:INFO=tcp:3306 \ -credential_file=../credFile.json', {shell: true});
         }else{
           kill(child.pid, function(err) {
             child = spawn('../cloud_sql_proxy -instances=YOUR:INSTANCE:INFO=tcp:3306 \ -credential_file=../credFile.json', {shell: true});
           });
         }

       }
       if(err.port == '3307'){
         if(!childRep){
           childRep = spawn('../cloud_sql_proxy -instances=YOUR:INSTANCE:INFO-replica=tcp:3307 \ -credential_file=../credFile.json', {shell: true});
         }else{
           // kill(1, 'SIGKILL', function(err) {
           kill(childRep.pid, function(err) {
             childRep = spawn('../cloud_sql_proxy -instances=YOUR:INSTANCE:INFO-replica=tcp:3307 \ -credential_file=../credFile.json', {shell: true});
           });
         }
       }
     }
});