Comments (16)
We have the same setup: a kubernetes pod having a web app container + cloudsql container. You can easily trap the sigterm signal the following way in your deployment:
command: ["/bin/bash", "-c", "trap 'sleep 15; exit 0' SIGTERM; /cloud_sql_proxy -dir=/cloudsql -instances=..."]
This delay will ensure the web app is shut down before the cloudsql proxy container (e.g. during rolling updates). Previously you'd need a custom container since the trap
command is not available in the scratch
image, but since the 1.09 release of the cloudsql proxy, they use alpine
as base, so it works out of the box.
from cloud-sql-proxy.
Simplest solution to stop TERM killing the proxy in Kubernetes is to setup container with:
command: ["/bin/sh", "-c", "/cloud_sql_proxy [options...]"]
^ This will cause the /bin/sh
is root process which in turns receives the signal from Kube. According to shell behaviour, it ignores any signals when there is process running inside the shell (ie it won't forward it).
But agree that ideal solution would be implement this inside the proxy:
- Receive TERM signal - set internal flag "stopping"
- In new connection handler, if "stopping" is set, refuse to connect
- In connection closed handler, if "stopping" is and this is the last open connection, exit the process
from cloud-sql-proxy.
If possible, I'd suggest writing a shell script that traps SIGTERM and emits a different signal to the cloudsql proxy.
from cloud-sql-proxy.
I'd like to see the proxy stop accepting new connections (but keep active ones alive). That way I can SIGTERM it and immediately start a new (version of the) proxy without interrupting service.
from cloud-sql-proxy.
A preStop hook execution will prevent the SIGTERM
signal from being sent until the script execution is completed. If you use command: ["/bin/bash", "-c", "sleep 15"]
as the command for the preStop hook you can stop shutdown.
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
Also you can add communication between containers using shared volumes https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/
Which could be used to instruct your preStop hook when to complete by creating a file at the end of your webserver shutdown inside the shared volume and making your preStop hook for cloudsql proxy wait for that file to exist before stopping using a sleep loop.
from cloud-sql-proxy.
A least for our use case a gracefull shutdown(stop listning for incoming connections and finish processing the current ones) on SIGTERM would solve the problem as we use connection pooling in our application.
from cloud-sql-proxy.
from cloud-sql-proxy.
Thanks for the thoughtful reply! I'm not sure how to tell Kubernetes not to send a SIGTERM, but I'll investigate a little and get back to you.
from cloud-sql-proxy.
@jesseshieh I am facing the same issue as you. Could you solve the issue? If yes, could you tell me how?
from cloud-sql-proxy.
I haven't solved it yet, but @hfwang's suggestion sounds good to me.
from cloud-sql-proxy.
It turned out that the entrypoint of my main container was not in exec format so SIGTERM was not transfered to nginx and it was functioning until SIGKILL stopped it finally.
from cloud-sql-proxy.
@park9140 or @mhindery were you guys able to get either of your solutions working? It seems like /bin/bash
and trap
are not in the gcr.io/cloudsql-docker/gce-proxy:1.09 image. sleep
is but setting a preStop
hook to just /bin/sleep 30
doesn't seem to work.
Also, I get a FailedPreStopHook
on the container when trying to sleep on the preStop
. I thought maybe the FailedPreStopHook
was related to this issue which seems to imply that the failure is noise and that the preStop hook does actually work. However, my sleep did not seem to work and the container was still sent SIGTERM immediately. UPDATE: turns out I should've done ["/bin/sh", "-c", "/bin/sleep 30"]
as my preStop command. This works as expected.
Ultimately I was able to get a working graceful shutdown by:
- making my own image from the gcr.io/cloudsql-docker/gce-proxy:1.09 base
- adding dumb-init to it as suggested here.
- changing my startup command to
/usr/local/bin/dumb-init --single-child --rewrite 15:0 /cloud_sql_proxy ...
to just completely drop the SIGTERM. - Using
["/bin/sh", "-c", "/bin/sleep 30"]
as apreStop
command for the cloudsql proxy container.
I have separate preStop hooks on my webapp containers that are correctly sleeping to drain connections so I originally thought I just needed cloud SQL proxy to not exit on SIGTERM. However, without the preStop
on cloudsql proxy, the container would still be killed shortly after the SIGTERM which would impact some requests. It originally appeared to be fixed in small tests but was not fully working for my use case until I added the preStop
I would much prefer a cleaner solution like you guys mentioned above. Am I missing something about how to get those working?
Thanks!
from cloud-sql-proxy.
from cloud-sql-proxy.
Thank you for the information. I had the same problem. I got it to work with sleep on preStop...
Anyway the solution does not seam very clean to me...
Maybe there is some way to get a environment variable or commandline parameter with a wait time before shuting down on SIGTERM ?
from cloud-sql-proxy.
from cloud-sql-proxy.
I'll close this thread and this will be resolved together with #128.
from cloud-sql-proxy.
Related Issues (20)
- Do not require strict order in env variables for connections HOT 2
- Add support for a lazy refresh
- pgbouncer + proxy with transaction pooling is slow HOT 23
- Is there any way to connect to MS SQL using domain credentials? HOT 7
- v2/tests: TestSQLServerAuthentication failed HOT 4
- Telemetry doesn't work with non-ADC
- v2/internal/proxy: TestCheckConnections failed HOT 2
- Connecting to cloud-sql using private-ip sometimes fails with a TLS handshake timeout HOT 7
- Availability of a Container Image on Google Artifact Registry HOT 2
- "Cloud SQL IAM service account authentication failed for user ..." intermittent errors when connecting to Postgres HOT 17
- CSQL_PROXY_ADDRESS requires an IP address but doesn't explicitly state so HOT 6
- 30s+ Hang When Using Manual Token Authentication HOT 7
- v2/internal/proxy: TestClientLimitsMaxConnections failed HOT 1
- v2/internal/proxy: TestClientCloseWaitsForActiveConnections failed HOT 1
- v2/internal/proxy: TestClientClosesCleanly failed HOT 1
- v2/internal/proxy: TestClosesWithError failed HOT 2
- v2/internal/proxy: TestClientConnCount failed HOT 2
- v2/internal/proxy: TestRunConnectionCheck failed HOT 2
- Automatic instance discovery isn't supported in v2 Proxy HOT 5
- Brief summary of the proposed feature
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloud-sql-proxy.