Comments (25)
I'm able to reproduce this error even when send_interrupt
is set to true. There seems to be a race condition when shutting down air using Ctrl+c.
When air is running the bin and a ctrl+c comes in cleanup() is executed. At this line in cleanup we call to have the bin stopped. When this call happens we aren't waiting for it to finish. https://github.com/cosmtrek/air/blob/master/runner/engine.go#L582-L586
The air program typically exits before it has the chance to stop the bin and get to https://github.com/cosmtrek/air/blob/master/runner/engine.go#L521-L522
from air.
This is also happening to me using ZSH. I have send_interrupt
set to true
and have tried a variety of configs but am still left with the process running after see you again~
.
As a bit of a work around until this is resolved, I have set the following config to automatically kill the process using a specific port:
post_cmd = ["lsof -i tcp:8080 | awk 'NR==2{print $2}' | xargs kill"]
from air.
I can confirm that the server is not killed even when send_interrupt
is set to true
from air.
I have a 5 second delay for connections to close on graceful shutdown. If you're listening for an interrupt then be sure to set a kill_delay as well otherwise air will close faster than your cleanup process.
# Send Interrupt signal before killing process (windows does not support this feature)
send_interrupt = true
# Delay after sending Interrupt signal
kill_delay = 5000000000 # nanosecond
from air.
Could folks having this problem give this a try?
from air.
@sethbrasile I have the same problem and #575 solved the issue for me π
from air.
The key here is to set send_interrupt
to true in your air.toml. only then will Ctrl+c behave as expected.
from air.
I had some issues with this as well. I tried your branch @sethbrasile, it helps but doesn't fix it entirely for me, there were times I had to manually kill the process. This code here could be more resilient: https://github.com/cosmtrek/air/blob/master/runner/util_linux.go#L12. It's ignoring some errors, and in my opinion the delay between SIGINT and SIGKILL should be the default, now the user has to know about the kill_delay
config. And as @peterldowns mentioned, this ideally shouldn't be a time.Sleep with a fixed interval being idle but rather a timeout. To implement this we could signal SIGINT and spin up a go routine checking if the PID still exists, once it doesn't we signal on the procKilledCh
from air.
@hamza72x I have similar settings but I'm seeing different behavior (the ^C is being ignored on MacOS).
When I run my server locally I see it exit from an interrupt as expected:
I0304 12:51:10.617018 47315 server.go:142] updating pod info
I0304 12:51:10.617172 47315 server.go:108] starting server at port 7442
^C
I0304 12:51:14.062000 47315 server.go:67] quitting server
E0304 12:51:14.062116 47315 server.go:59] http: Server closed
Doing the same thing with air i see:
/ /\ | | | |_)
/_/--\ |_| |_| \_ v1.51.0, built with Go go1.22.0
watching .
watching cmd
watching cmd/web
!exclude dist
watching internal
watching internal/api
!exclude node_modules
watching pkg
!exclude public
!exclude src
!exclude tmp
building...
running...
I0304 12:52:50.795311 48146 server.go:142] updating pod info
I0304 12:52:50.795335 48146 server.go:108] starting server at port 7442
^C
cleaning...
see you again~
but when I check lsof i still see the process running and bound to a port (and it still logs to my local terminal):
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
web 48146 me 3u IPv6 0xba380d8c0d1589c5 0t0 TCP *:7442 (LISTEN)
here is my config
testdata_dir = "testdata"
tmp_dir = "tmp"
[build]
args_bin = []
bin = "./web"
cmd = "go build -o ./web ./cmd/web/main.go"
delay = 1000
exclude_dir = ["dist", "assets", "tmp", "vendor", "testdata", "node_modules", "src", "public"]
exclude_file = []
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
full_bin = ""
include_dir = []
include_ext = ["go", "tpl", "tmpl", "html"]
include_file = []
kill_delay = "0s"
log = "build-errors.log"
poll = false
poll_interval = 0
post_cmd = []
pre_cmd = []
rerun = false
rerun_delay = 500
send_interrupt = true
stop_on_error = true
[color]
app = ""
build = "yellow"
main = "magenta"
runner = "green"
watcher = "cyan"
[log]
main_only = false
time = false
[misc]
clean_on_exit = false
[screen]
clear_on_rebuild = false
keep_scroll = true
from air.
signal.Notify(quit, os.Interrupt)
Be sure to capture syscall.SIGTERM also. systemd/docker uses that signal for stop and restart. Save yourself some headaches on deployment.
signal.Notify(quit, os.Interrupt, syscall.SIGTERM)
from air.
@taylow Improved on your solution a little bit.
I ensure I reap the process with a given listening port before every reload and when you quit air
.
post_cmd = [
"lsof -i :8080 -sTCP:LISTEN | sed 1d | awk '{print $2}' | xargs kill -9"
]
pre_cmd = [
"lsof -i :8080 -sTCP:LISTEN | sed 1d | awk '{print $2}' | xargs kill -9"
]
This part of the lsof
command is particularly important as it prevents your browser or other tools with open or establishing connections to your server also getting reaped.
lsof -sTCP:LISTEN
Hope it helps and I hope this will be fixed soon so we don't have to use a workaround. β€οΈ
from air.
@peterldowns Are you on the latest air also? My air -v is v1.51.0, built with Go go1.22.2.
Is there sample code I can try? Are you doing anything special to build?
I want to see if I can replicate the issue or if there is a service that isn't closing correctly.
from air.
@figuerom16 I'll set up a minimal reproduction case and share it here within the next day or so. Sorry for not doing so earlier, I realize it's hard to debug without an easy way to reproduce the problem. I'm using air installed via nix, but it's 1.51.0 built with go1.22.1. I'm on macOS Monterey version 12.5.1 on a M1 pro:
β― air -v
__ _ ___
/ /\ | | | |_)
/_/--\ |_| |_| \_ (devel), built with Go go1.22.1
β― which air
/nix/store/1w5403arzypxxb1s2mlaylxmnmlvwjq6-air-1.51.0/bin/air
from air.
This repo is a reproducible example, if anyone needs: https://github.com/sa-/goth-counter
from air.
I had the same issue and was able to permanently fix it by adding:
post_cmd = ["pkill <BINARY_NAME>"]
from air.
I am fairly new to the language with 3 months experience and I diagnosed that the context.WithTimeout is causing it. I tried it with context.TODO() and it works fine now
from air.
Had similar issue, just use pkill
and update your .air.toml
like this.
root = "."
testdata_dir = "testdata"
tmp_dir = "tmp"
[build]
bin = "./tmp/my-api-go"
cmd = "pkill my-api-go; go build -o ./tmp/my-api-go ."
delay = 1000
exclude_dir = ["assets", "tmp", "vendor", "testdata", "node_modules"]
exclude_file = []
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
full_bin = ""
include_dir = []
include_ext = ["go", "tpl", "tmpl", "html"]
kill_delay = "0s"
log = "build-errors.log"
send_interrupt = false
stop_on_error = true
[color]
app = ""
build = "yellow"
main = "magenta"
runner = "green"
watcher = "cyan"
[log]
time = false
[misc]
clean_on_exit = false
[screen]
clear_on_rebuild = false
from air.
I was having the same issue as you. I have a docker container running a go service where I listen for SIGINT and SIGTERM after starting the server so I can gracefully shutdown. I was able to resolve with the following:
[build]
...
send_interrupt = true
kill_delay = 1
If you don't set something for kill delay it seems like Air eats the signal.
from air.
Just writing in to say that even with send_interrupt=true
and kill_delay=1
, I still see intermitten failure to send the ctrl-c
interrupt to my program. I'm using the pre_cmd
and post_cmd
hacks shared above (thank you!) but it's annoying.
from air.
Referring to the example toml https://github.com/cosmtrek/air/blob/master/air_example.toml
kill_delay operates in nanoseconds! With a value of 1 you're giving your program maybe CPU 4 clock cycles to close up your connections and program.
At least set your delay to 100 milliseconds | kill_delay = 100000000
Just timing my DB and app close takes nearly 500 microseconds so 100 milliseconds is pretty safe for Testing/Development.
EDIT: Here is my gracefull shutdown portion for reference.
...
// Server Run
go func() {
if err := server.ListenAndServe(); err != nil {
log.Println(err)
}
}()
log.Println("Server Running: http://127.0.0.1" + port)
// Graceful shutdown
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
<-c
start := time.Now()
log.Println("Gracefully shutting down...")
server.Shutdown(ctx)
data.Close()
log.Printf("Gracefully closed %s.", time.Since(start))
Output:
main.go has changed
building...
2024/04/24 13:39:17 Gracefully shutting down...
2024/04/24 13:39:17 http: Server closed
2024/04/24 13:39:17 Database closed
2024/04/24 13:39:17 Gracefully closed 364.905Β΅s
running...
2024/04/24 13:39:18 Server Running: http://127.0.0.1:3000
from air.
@figuerom16 I've tried delay values of 5000000000
as well as "5000ms"
but it simply does not delay.
from air.
@sethbrasile I can confirm that this fixed the issue for me as well. I tested by cloning your fork, building the air
binary, then updating my .air.toml
to comment out the pkill
shenanigans, and set kill_delay = "5000ms"
. Now, when I hit ctrl-c, air
propagates the signal to my server, and the server has time to exist cleanly. Hooray!
One suggestion β can you make it so that if the backend server has terminated, air
will stop waiting? For instance:
- If the server takes ~1,000ms to exit, and
kill_delay = 5,000ms
, then air should exit after ~1,000ms, because the server has exited and there is no reason for air to continue waiting. - If the server takes ~20,000ms to exit, and
kill_delay = 5,000ms
, then air should exit after 5,000ms because that is the maximum amount of time it should wait.
from air.
Also, thank you for the fix!
from air.
pkill
uses SIGTERM and also Docker kills containers with SIGTERM. Maybe we can switch from SIGINT to SIGTERM, which is a signal most likely applications are handling already, and a default timeout before SIGKILL
from air.
Wild guess, but it seems that upgrading MacOS from 14.4.1 to 14.5 seems to have resolved this for me. Can anyone else confirm?
Previously I reproduced this problem of failing to kill process with SIGINT/SIGTERM with both air and modd, so I don't think air is to blame specifically.
from air.
Related Issues (20)
- εε¨ζε»Ίε€±θ΄₯ HOT 1
- ζ―ζε€δΈͺε―ζ§θ‘ζδ»Ά HOT 1
- Make pre_cmd background forkable HOT 3
- Air not being installed in zsh path on macOS HOT 6
- building memos not use docker in windows10
- i want to install the imported packages with air reloading
- Air always assumes that there is binary build target HOT 2
- zsh: permission denied:
- Hide logo or show custom text
- post_cmd on Windows with paths with spaces doesn't seem to work HOT 1
- pre_cmd fails HOT 3
- Hot Reload Not Working on Rancher Desktop
- Hot Reload not working for changes on main.go HOT 2
- Add Homebrew formula
- Air doesn't seem to be working when there's whitespace in the path.
- Disable tmp directory HOT 1
- panic: unaligned 64-bit atomic operation in windows HOT 3
- panic when shutting down with SIGINT signal if proxy is enabled HOT 8
- bug: live reload do not work for tmpl files HOT 6
- Using the `air` cli on Windows breaks when providing nested paths to `build.bin`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from air.