Release all connections if too many files are opened about go-any-proxy

[louchenyao]

Thanks for your advice!

I figured it out that I truly need is to close any_proxy when err happens in accepting

I don't know why there are so many new connections. Probably my wrong configurations in iptables create a "self-loop". It happens rarely so restarting is good to me. Thank you again for your time!

[ryanchapman]

In your case, how many file descriptors are open when they become exhausted? Also, what is your load profile? How many connections per second does any_proxy see? What type of hardware are you running it on (number of CPUs, memory)?

[louchenyao]

Sorry I didn't monitor that information like reqs/sec.

I tried to monitor /porc/{any_proxy_pid}/fd. But once fds are suddenly exhausted, I cannot even run ls /porc/{any_proxy_pid}/fd so it is hard to track #fd, which is why I hope any_proxy could prevent this happening.

I have run ulimit -n 65535 before running any_proxy.

I run it on a single-core x64 CPU with 512 MB memory. The any_proxy repeatedly logs the following lines when it runs at 100% CPU:

2020/02/06 00:32:05 any_proxy.go:360: : INFO : Error accepting connection: accept tcp [::]:3140: accept4: too many open files
2020/02/06 00:32:05 any_proxy.go:360: : INFO : Error accepting connection: accept tcp [::]:3140: accept4: too many open files

[ryanchapman]

any_proxy (really the kernel) is denying connections and instead logging that it couldn't accept an incoming connection because it was out of resources.

A single core with half a gig of memory isn't very much, especially if the machine is seeing any type of load. You could try increasing the number of available file descriptors with ulimit -n 250000, but I think that is going to just cause more problems. Each incoming connection needs memory (I can't remember how much off the top of my head though).

When I originally wrote any_proxy, I load tested it on a very big machine, like 72 cores and 100 GB of memory, and was able to get to 10k req/sec. Could you try adding more cores and memory to the machine and try again?

[louchenyao]

Yep, It did deny new connections. I think I should change the title to better express my problem. My bad.

Once it used up all fds, the os becomes "inoperable" because other processors like ssh cannot start new connections. If it could restart itself to release all connections instead of eating up all CPU to retry the "accept"(some buggy clients may keep many long TCP connections), at least the server can be operatable.

[louchenyao]

I forked and implemented this feature. My implementation will track all connections, and release the oldest ones if there are too many.

If you are interested in this I can submit a pull request.

[ryanchapman]

The normal way this is handled in the unix world is by limiting incoming connections via the ulimit -n command. In your case, on a small server (1 CPU, 0.5GB of memory), I would recommend keeping the ulimit for file-max at the default of 1024. I suspect that will make it stable (assuming just a few clients are connecting to it).

That said, if your fork works for your use case, great. I'm not sure that the any_proxy community would really benefit from such an approach though. It is designed for high load and running on large machines. There are some fortune 100 companies that have been pushing 10k/req through it for years with no issues. Some of them are running it on 16 core VMs with 16GB of memory.