test_protocol unittest failures on Windows about websockets

[aaugustin]

Great, thanks for reporting these. I also noticed some failures when running tests under tox. (I'm not sure why this makes a difference.)

I'm a bit busy today, I'll try to follow up as soon as possible.

[SzieberthAdam]

The ailures above were produced by my work environment, an old IBM Thinkcentre with 2.8 Ghz Celeron CPU. Current lines are 488, 366, and 382. This PC produces the failures above constantly.

However, I was also tested it with my (also old) MSI laptop (AMD Athlon 64 X2 Dual-Core TK-57) which produces inconsistent results.

[adam@szamsi websockets]$ python -m unittest
...........Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: /usr/lib/python3.4/asyncio/tasks.py:332: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 8642, 0, 0), raddr=('::1', 37612, 0, 0)>
  return (yield from _wait(fs, timeout, return_when, loop))
/usr/lib/python3.4/asyncio/tasks.py:332: ResourceWarning: unclosed <socket.socket fd=9, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37612, 0, 0)>
  return (yield from _wait(fs, timeout, return_when, loop))
/usr/lib/python3.4/asyncio/tasks.py:332: ResourceWarning: unclosed <socket.socket fd=12, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 8642, 0, 0), raddr=('::1', 37613, 0, 0)>
  return (yield from _wait(fs, timeout, return_when, loop))
/usr/lib/python3.4/asyncio/tasks.py:332: ResourceWarning: unclosed <socket.socket fd=11, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37613, 0, 0)>
  return (yield from _wait(fs, timeout, return_when, loop))
/usr/lib/python3.4/asyncio/tasks.py:332: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 8642, 0, 0), raddr=('::1', 37614, 0, 0)>
  return (yield from _wait(fs, timeout, return_when, loop))
/usr/lib/python3.4/asyncio/tasks.py:332: ResourceWarning: unclosed <socket.socket fd=13, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37614, 0, 0)>
  return (yield from _wait(fs, timeout, return_when, loop))
...Exception ignored in: Exception ignored in: /usr/lib/python3.4/email/message.py:122: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 8642, 0, 0), raddr=('::1', 37615, 0, 0)>
  self.policy = policy
/usr/lib/python3.4/email/message.py:122: ResourceWarning: unclosed <socket.socket fd=9, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37615, 0, 0)>
  self.policy = policy
......Exception ignored in: /usr/lib/python3.4/encodings/ascii.py:26: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 8642, 0, 0), raddr=('::1', 37622, 0, 0)>
  return codecs.ascii_decode(input, self.errors)[0]
.........Exception ignored in: /usr/lib/python3.4/email/message.py:122: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37632, 0, 0), raddr=('::1', 8642, 0, 0)>
  self.policy = policy
....Exception ignored in: Exception ignored in: /usr/lib/python3.4/email/feedparser.py:200: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37634, 0, 0), raddr=('::1', 8642, 0, 0)>
  msg = self._factory(**self._factory_kwds())
/usr/lib/python3.4/email/feedparser.py:200: ResourceWarning: unclosed <ssl.SSLSocket fd=11, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37635, 0, 0), raddr=('::1', 8642, 0, 0)>
  msg = self._factory(**self._factory_kwds())
...Exception ignored in: /usr/lib/python3.4/asyncio/base_events.py:579: ResourceWarning: unclosed <ssl.SSLSocket fd=10, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 37633, 0, 0), raddr=('::1', 8642, 0, 0)>
  yield from tasks.wait(fs, loop=self)
...Exception ignored in: /usr/lib/python3.4/selectors.py:62: ResourceWarning: unclosed <ssl.SSLSocket fd=10, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('::1', 8642, 0, 0), raddr=('::1', 37642, 0, 0)>
  return self._selector._fd_to_key[fd]
...........................................FF..........................................F.................................
======================================================================
FAIL: test_close_timeout_before_connection_lost (websockets.test_protocol.ClientTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 505, in test_close_timeout_before_connection_lost
    self.assertTrue(self.after.cancelled())
AssertionError: False is not true

======================================================================
FAIL: test_close_timeout_before_eof_received (websockets.test_protocol.ClientTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 488, in test_close_timeout_before_eof_received
    self.assertFalse(self.before.cancelled())
AssertionError: True is not false

======================================================================
FAIL: test_close_handshake_timeout (websockets.test_protocol.ServerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 366, in test_close_handshake_timeout
    self.assertFalse(self.before.cancelled())
AssertionError: True is not false

----------------------------------------------------------------------
Ran 160 tests in 4.297s

This was a first-time run. For the subsequent test, when I wait some time, only line 505 fails. However, when I start another unittest immediately, 366 seems to fail sometimes aswell.

I switch to the Windows OS on the MSI to test there too...

Hmm...

adam@SZAMSI e:\websockets
> python -m unittest
...............................Exception ignored in: Exception ignored in: Exception ignored in: Exception ignored in: C:\Python34\lib\asyncio\tasks.py:72: ResourceWarning: unclosed <socket.socket fd=652, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 8642, 0, 0), raddr=('::1', 49271, 0, 0)>
  super().__init__(loop=loop)
C:\Python34\lib\asyncio\tasks.py:72: ResourceWarning: unclosed <ssl.SSLSocket fd=368, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 49301, 0, 0), raddr=('::1', 8642, 0, 0)>
  super().__init__(loop=loop)
C:\Python34\lib\asyncio\tasks.py:72: ResourceWarning: unclosed <ssl.SSLSocket fd=256, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 49304, 0, 0), raddr=('::1', 8642, 0, 0)>
  super().__init__(loop=loop)
C:\Python34\lib\asyncio\tasks.py:72: ResourceWarning: unclosed <ssl.SSLSocket fd=612, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 49307, 0, 0), raddr=('::1', 8642, 0, 0)>
  super().__init__(loop=loop)
................................................F..FF..........................................F.Exception ignored in: C:\Python34\lib\unittest\mock.py:515: ResourceWarning: unclosed <ssl.SSLSocket fd=388, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 8642, 0, 0), raddr=('::1', 49331, 0, 0)>
  self._mock_side_effect = value
F...............................
======================================================================
FAIL: test_client_close (websockets.test_protocol.ClientTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "e:\websockets\websockets\test_protocol.py", line 458, in test_client_close
    self.assertConnectionClosed(1000, 'because.')
  File "e:\websockets\websockets\test_protocol.py", line 79, in assertConnectionClosed
    self.assertEqual(self.protocol.close_code, code)
AssertionError: 1006 != 1000

======================================================================
FAIL: test_close_timeout_before_connection_lost (websockets.test_protocol.ClientTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "e:\websockets\websockets\test_protocol.py", line 506, in test_close_timeout_before_connection_lost
    self.assertFalse(self.before.cancelled())
AssertionError: True is not false

======================================================================
FAIL: test_close_timeout_before_eof_received (websockets.test_protocol.ClientTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "e:\websockets\websockets\test_protocol.py", line 488, in test_close_timeout_before_eof_received
    self.assertFalse(self.before.cancelled())
AssertionError: True is not false

======================================================================
FAIL: test_close_handshake_timeout (websockets.test_protocol.ServerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "e:\websockets\websockets\test_protocol.py", line 366, in test_close_handshake_timeout
    self.assertFalse(self.before.cancelled())
AssertionError: True is not false

======================================================================
FAIL: test_close_timeout_before_connection_lost (websockets.test_protocol.ServerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "e:\websockets\websockets\test_protocol.py", line 382, in test_close_timeout_before_connection_lost
    self.assertFalse(self.before.cancelled())
AssertionError: True is not false

----------------------------------------------------------------------
Ran 160 tests in 7.623s

FAILED (failures=5)

At least these failures seem to be consistent on that environment.

Hard to test asyncio code, that is for sure. :)

[SzieberthAdam]

Now I noticed I do testing while the repo is acively changing. I get back later.

[aaugustin]

Yes, I just pushed a bunch of changes I've been working on yesterday and today. (I have limited Internet connectivity.) But I don't think they help for this issue.

I can reproduce test_close_timeout_before_connection_lost reliably. I'm going to look into it. I haven't seen other failures regularly.

If you're running the tests on a slow machine, try increasing the value of MS in websockets/test_protocol.py and see if that helps.

[aaugustin]

I took a closer look at your report. There are several issues.

1. ClientTests.test_close_timeout_before_connection_lost was buggy. I'm working a fix, but I haven't figured it out completely yet.
2. Failures on self.assertFalse(self.before.cancelled()) are probably explained by the difference between your hardware / OS and mine. These tests check that timeouts are applied correctly by asserting that some coroutines complete in a given time. Like I said above, changing MS should make them pass. I'd like to keep MS set to 1 millisecond by default because that makes the tests run very fast :-) However I could allow overriding it with an environment variable for convenience (and if I ever want to run these tests on some cloud CI service).
3. ClientTests.test_client_close could be a different syndrom of the same cause root cause. It looks like the coroutine scheduled at 3ms runs before the sequence of actions triggered at 1ms completes.

Would you mind pulling the latest commit to master, setting MS = 0.01 in websockets/test_protocol.py and letting me know if you still see any failures? (If you have tox installed, you can simply run tox.)

[SzieberthAdam]

Sure, I'll do that today afternoon/evening.

[SzieberthAdam]

Done. Conclusions:

1. Yep, the remaining failure is produced by lines 505-506, mostly 505. However, when I set the MS lower (0.0005) then line 506 produces the failure. Are not the two rows somehow in the opposite order? Increasing MS even with insane amount does not solve it.
2. test_close_timeout_before_eof_received(), test_close_handshake_timeout(), and test_close_timeout_before_connection_lost() all fail at assertFalse(). By setting MS to 0.003 they pass even on my slowest PC. (I usually like to develop on old PCs. Easier to recognise bad code by noticing slowness.) The time of the testing increases by 0.15 s after the global MS adjustment. What I would do is to adjust the delays in these particular tests a bit. I will do that gladly if you agree Aymeric.
3. Yes, for some reason, test_client_close() works the same like (1.), although it passes at the end. At MS=0.0001 it fails at line 460, at MS=0.0005 it fails at line 458. However, at MS=0.001 it passes (on my laptop/ARCH; the Thinkcentre needs MS=0.002 to not fail at 458).

[aaugustin]

Thanks for the feedback.

I just had an idea to make the tests both fast and robust: instead of using self.loop.call_later(X * MS, do_stuff) with incremental values of X, I can just use self.loop.call_soon(do_stuff). asyncio guarantees that functions scheduled with call_soon are executed in order.

I'm working on a patch. Not only does this makes the tests look much nicer, but they should also be more deterministic and faster. Then I can increase the value of MS when I test timeouts without making the whole test suite proportionally slower.

Let's see where we stand once I've completed this refactoring!

[SzieberthAdam]

Good idea! I wasn't aware of this benefit of call_soon() until now.

[aaugustin]

I just pushed a series of commits that should improve the situation significantly.

Feel free to test again and report your findings at your convenience. Thanks!

[aaugustin]

Tentatively closing as fixed.

[SzieberthAdam]

At commit 9bdfb81 on my Windows environment:

adam@SZAMSI E:\websockets
> python -m unittest
...........................................................................................................F.....F........................F.......F.....F.................................
======================================================================
FAIL: test_close_handshake_timeout (websockets.test_protocol.ClientCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\websockets\websockets\test_protocol.py", line 581, in test_close_handshake_timeout
    self.loop.run_until_complete(self.protocol.close(reason='close'))
  File "C:\Python34\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "E:\websockets\websockets\test_protocol.py", line 167, in assertCompletesWithin
    dt, min_time, "Too fast: {} < {}".format(dt, min_time))
AssertionError: 0.01599999999962165 not greater than or equal to 0.019 : Too fast: 0.01599999999962165 < 0.019

======================================================================
FAIL: test_eof_received_timeout (websockets.test_protocol.ClientCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\websockets\websockets\test_protocol.py", line 593, in test_eof_received_timeout
    self.loop.run_until_complete(self.protocol.close(reason='close'))
  File "C:\Python34\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "E:\websockets\websockets\test_protocol.py", line 167, in assertCompletesWithin
    dt, min_time, "Too fast: {} < {}".format(dt, min_time))
AssertionError: 0.0 not greater than or equal to 0.009000000000000001 : Too fast: 0.0 < 0.009000000000000001

======================================================================
FAIL: test_server_close_race_with_failing_connection (websockets.test_protocol.ClientCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\websockets\websockets\test_protocol.py", line 609, in test_server_close_race_with_failing_connection
    self.assertOneFrameSent(*self.server_close)
  File "E:\websockets\websockets\test_protocol.py", line 147, in assertOneFrameSent
    self.assertEqual(self.last_sent_frame(), Frame(fin, opcode, data))
AssertionError: Frame(fin=True, opcode=8, data=b'\x03\xe8client') != Frame(fin=True, opcode=8, data=b'\x03\xe8server')

======================================================================
FAIL: test_client_close_race_with_failing_connection (websockets.test_protocol.ServerCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\websockets\websockets\test_protocol.py", line 473, in test_client_close_race_with_failing_connection
    self.assertOneFrameSent(*self.client_close)
  File "E:\websockets\websockets\test_protocol.py", line 147, in assertOneFrameSent
    self.assertEqual(self.last_sent_frame(), Frame(fin, opcode, data))
AssertionError: Frame(fin=True, opcode=8, data=b'\x03\xe8server') != Frame(fin=True, opcode=8, data=b'\x03\xe8client')

======================================================================
FAIL: test_close_handshake_timeout (websockets.test_protocol.ServerCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\websockets\websockets\test_protocol.py", line 458, in test_close_handshake_timeout
    self.loop.run_until_complete(self.protocol.close(reason='close'))
  File "C:\Python34\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "E:\websockets\websockets\test_protocol.py", line 167, in assertCompletesWithin
    dt, min_time, "Too fast: {} < {}".format(dt, min_time))
AssertionError: 0.0 not greater than or equal to 0.009000000000000001 : Too fast: 0.0 < 0.009000000000000001

----------------------------------------------------------------------
Ran 186 tests in 3.962s

FAILED (failures=5)

All tests passes on same PC in ARCH Linux environment.

However my Thinkcentre produces some of the filures too on ARCH Linux:

[adam@sza8215 websockets]$ python -m unittest
...........................................................................................................F.....F......................................F.................................
======================================================================
FAIL: test_close_handshake_timeout (websockets.test_protocol.ClientCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 581, in test_close_handshake_timeout
    self.loop.run_until_complete(self.protocol.close(reason='close'))
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 169, in assertCompletesWithin
    dt, max_time, "Too slow: {} >= {}".format(dt, max_time))
AssertionError: 0.026171876001171768 not less than 0.025 : Too slow: 0.026171876001171768 >= 0.025

======================================================================
FAIL: test_eof_received_timeout (websockets.test_protocol.ClientCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 593, in test_eof_received_timeout
    self.loop.run_until_complete(self.protocol.close(reason='close'))
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 169, in assertCompletesWithin
    dt, max_time, "Too slow: {} >= {}".format(dt, max_time))
AssertionError: 0.015940716002660338 not less than 0.015 : Too slow: 0.015940716002660338 >= 0.015

======================================================================
FAIL: test_close_handshake_timeout (websockets.test_protocol.ServerCloseTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 458, in test_close_handshake_timeout
    self.loop.run_until_complete(self.protocol.close(reason='close'))
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/adam/github/notmy/websockets/websockets/test_protocol.py", line 169, in assertCompletesWithin
    dt, max_time, "Too slow: {} >= {}".format(dt, max_time))
AssertionError: 0.016229974004090764 not less than 0.015 : Too slow: 0.016229974004090764 >= 0.015

----------------------------------------------------------------------
Ran 186 tests in 4.223s

FAILED (failures=3)

[aaugustin]

The second set of failures is clearly caused by the old Thinkcentre being a bit slow. With WEBSOCKETS_TESTS_TIMEOUT_FACTOR=2 python -m unittest the tests should pass. Fewer tests depend on fixed timeouts now, so I could increase their duration without harming the test suite's overall run time.

[aaugustin]

I would need access to a Windows machine to reproduce your first set of failures. Two causes come to mind:

• the clock used by asyncio on Windows not having a very good resolution
• IOCP (overlapped I/O) doing smart things I do not expect

[SzieberthAdam]

Confirmed working with WEBSOCKETS_TESTS_TIMEOUT_FACTOR=2 environment variable on slow PC.

[aaugustin]

asyncio's clock relies on time.monotonic(). It is implemented with GetTickCount on Windows which has ~16ms resolution. That's why some tests fail with timings rounded to 0ms or 16ms.

[aaugustin]

With the commit I just pushed, I don't see failures on Windows anymore.

[SzieberthAdam]

Confirmed fixed.

[aaugustin]

Success \o/ Thanks!