Random SIGPIPE error when writing to a closed socket on a surge of requests about mitmproxy

[changsin]

Actually, after some extensive experiments and use, SIG_IGN seems to fix the issue.

Here is the PR:
#6764

[mhils]

Can you please verify if that still reproduces with the latest mitmproxy release, and also provide the full traceback including the SIGPIPE error message?

[changsin]

I upgraded to the latest build and it still repros.
The offending line is still the same. Just the line numbers are different.
Would it make sense to put a lock for a fix?

File "asyncio/runners.py", line 44, in run
File "asyncio/base_events.py", line 636, in run_until_complete
File "asyncio/base_events.py", line 603, in run_forever
File "asyncio/base_events.py", line 1909, in _run_once
File "asyncio/events.py", line 80, in _run
File "mitmproxy/proxy/server.py", line 294, in handle_connection
File "mitmproxy/proxy/server.py", line 407, in server_event
File "asyncio/streams.py", line 325, in write
File "asyncio/selector_events.py", line 924, in write

mitmproxy version info:

Mitmproxy: 10.2.4
Python: 3.10.11
OpenSSL: OpenSSL 3.2.1 30 Jan 2024
Platform: macOS-14.4-arm64-arm-64bit

Here is how we are capturing the SIGPIPE signal
###Received signal: 13, printing stack trac

def signal_handler(signal_received, frame):
    logger.error(f"###Received signal: {signal_received}, printing stack trace:")
    traceback.print_stack(frame)


def hookup_signal_handler():
    signal.signal(signal.SIGPIPE, signal_handler)

[mhils]

After looking at this for a bit -- I'm all but an expert on unix signals --, it sounds like mitmproxy should run signal(SIGPIPE, SIG_IGN) on startup. Can you confirm that this would fix the issue for you (for testing you can run that from an addon)? If yes, can you please send a PR that add this call here? Thanks!

[changsin]

You mean like this, right?
signal.signal(signal.SIGINT, _sigint)
signal.signal(signal.SIGTERM, _sigterm)
signal.signal(signal.SIGPIPE, SIG_IGN) <-- added

This just ignores SIGPIPE signal, right? Would it make the bad connection stay bad?
How is it different from the signal handler that I had earlier?

How about adding this to server.py? In the signal_handler, we close the bad connection.
This seems to work better in our internal testing.

        # workaround for https://bugs.python.org/issue40124 / https://bugs.python.org/issue29930
        self._drain_lock = asyncio.Lock()
        signal.signal(signal.SIGPIPE, self.signal_handler)

    def signal_handler(self, signal_received, frame):
        self.log(f"###Received signal1: {signal_received}, printing stack trace:", logging.ERROR)
        traceback.print_stack(frame)
        self.log("client kill connection")
        writer = self.transports.pop(self.client).writer
        assert writer
        writer.close()

[mhils]

This seems to work better in our internal testing.

Can you clarify by which metric this works better? The code you posted seems to work as well, but reassigning the signal handler on every write seems unnecessarily costly.

FYI If you run git diff > out.txt and then paste the contents of out.txt here, we get a much nicer output that's easier to interpret due to filenames and line numbers. :)

[hans-at-strac]

We added signal.signal(signal.SIGPIPE, SIG_IGN) but that did not resolve the issue.

The symptoms on my machine:

• If you open Slack the next day, Slack is not able to connect
• Intermittently, some websites won't load up but after 30 seconds, it will automatically retry and connect.

[mhils]

@hans-at-strac: To confirm, the patch posted by @changsin does definitely fix the issue, but just setting SIG_IGN doesn't?

We're seeing one error message here (SIGPIPE) and some particular networking issues. But I'd like to make sure that there also is causality between these two.

Can you send SIGUSR1 / SIGUSR2 to the mitmproxy process when this Slack issue occurs? Ideally on an otherwise silent network.