bad performance: sending stdin from client to server about nailgun

[mihaip]

@jvican see mihaip/nailgun@a738fd6 for the code and a simple benchmark. If you run it, you'll see:

$ python -m pynailgun.test_ng TestNailgunConnection.test_echo_latency
NailgunConnection: 965.1ms (min: 507ms, max: 1058.3ms)
BytesInputNailgunConnection: 4.2ms (min: 2.7ms, max: 6.8ms)
.
----------------------------------------------------------------------
Ran 1 test in 9.888s

OK
NailgunConnection: 962.9ms (min: 503.2ms, max: 1062.6ms)
BytesInputNailgunConnection: 3.9ms (min: 2.5ms, max: 8.6ms)
.
----------------------------------------------------------------------
Ran 1 test in 9.880s

OK

[sbalabanov-zz]

not on master yet, hopefully will go out this week

[stuhood]

I suspect that this has something to do with the fact that both the C client and Python client (and thus probably also the Java server) have diverged from the spec and now require a "start reading stdin" chunk after every stdin chunk is sent from the client. The constant ping-ponging certainly doesn't help performance.

[mihaip]

That's been my observation too. I was seeing response times with a bimodal distribution of either 1000 or 500ms, even though the actual work on the Java side took ~30ms.

The problem was that the Python Nailgun client reads stdin in a separate thread, which races with the main thread

1 [main thread]: _process_next_chunk calls select()
2 [stdin thread]: wait on ready_to_send_condition
3 [java process]: sends CHUNKTYPE_SENDINPUT
4 [main thread]: select() returns, process_nailgun_stream is called, which notifies ready_to_send_condition once it determines that CHUNKTYPE_SENDINPUT was sent
5 [main thread]: calls _check_stdin_queue, but the stdin thread hasn't finished reading stdin, so there's nothing there
6 [stdin thread]: ready_to_send_condition is done waiting, stdin is read, data is added to the stdin_queue
7 [main thread]: next iteration of _send_command_and_read_response, select() is called again. The java process is still waiting to read from stdin, so select() blocks until the timeout fires (normally 500ms)
8 [main thread]: _check_stdin_queue finally has data, so the stdin can be sent to the java process

Steps 5 and 6 happen concurrently, but because we don't start reading stdin until after the condition notification happens, we most likely will not enqueue anything before the check happns.

In our case, we knew exactly what we want to send to the java process, we shouldn't need to go through a reader at all. Therefore we introduced a simpler BytesInputNailgunConnection that directly sends a bytes instance to the java process without needing a separate thread at all.

[jvican]

@mihaip Can you share more about the BytesInputNailgunConnection? I'm also experiencing this issue downstream and it's quite annoying, since I was using stdin/stdout for LSP.

[ttsugriy]

cc @sbalabanov

[sbalabanov-zz]

This should be fixed with Nailgun Python client 0.9.3

[mihaip]

@sbalabanov is that version on master (or otherwise publicly accessible -- there are no recent commits in https://github.com/facebook/nailgun/commits/master/pynailgun)? I'm not seeing any performance difference in my benchmark.

[jvican]

PR is merged now. I'll try it out downstream to see if it works for me. @sbalabanov I assume this has already been integrated with buck?

[sbalabanov-zz]

Yes it is integrated with buck but not a part of any official release yet