Unable to StreamWriter.drain() about websockets

For information, I came across this problem using websockets==3.2 while I was trying to use many "await websocket.send(data)" in parallel. WebSocketCommonProtocol.send() calls asyncio.StreamWriter.drain() which apparently must not be called twice in parallel.

I was able to reproduce it without websockets and opened an issue here: http://bugs.python.org/issue29930
I think it might be interesting to follow the evolution of this issue and make changes in websockets if necessary.

@aaugustin

[aaugustin]

I have no idea. Unfortunately, my crystal ball is momentarily out of order, so I can't read the code you have carefully omitted from this bug report.

[aaugustin]

Regardless of how you're running into this problem, make sure you're using the latest version of asyncio or try your code with Python 3.4, as some bugs have been fixed recently in this area.

[aaugustin]

Thanks @metathink, great investatigation!

[RemiCardona]

I'm glad to see this being reopened. I've been investigating this on and off for the past 2 weeks and came to the same conclusion: drain is nowhere near "thread"/task safe. And because websockets may call write_frame independently (IOW: in another Task) of user code through the PONG reply, there's a chance things may break.

I've been a able to reliably force the bug using https://github.com/mariusae/trickle (with a few local fixes to make it work with CLOEXEC sockets that python uses) to slow down network write speeds, which forces the sender to fill up the write buffer, thus revealing the bug.

The patch above simply adds a lock around drain() and I haven't been able to get this code to fail.

I've also been reading njs's "post async/await" post and the more I think about it, a module like websocket (and many others, really) could probably do away with all the whole transports._FlowControlMixin: writes could/should be coroutines that "block" until frame data has been flushed to kernel/libc buffers. The userspace/asyncio buffering is counterproductive. The drain fail being just the tip of the iceberg. But that's work for another PR.

[aaugustin]

Thanks for the patch, let's do that. Even if the issue eventually gets fixed in Python, we need to work around it in versions of Python where it exists.

[aaugustin]

See #170 for the more general considerations.

[aaugustin]

@RemiCardona Once you merge your patch, it would be a good time to make a release, wouldn't it?

[aaugustin]

I released version 3.3 with this change.