IndexError due to empty args in OpenSSL.SSL.ZeroReturnError handling in Twisted about twisted

[glyph]

Not sure if this would be the relevant test for this issue. Not too familiar with the inner-workings of twisted. https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/test/test_tls.py#L1008

FYI, if you hit y on any source page on Github, you get a link more like this

twisted/src/twisted/protocols/test/test_tls.py

Line 1008 in c465c46

def test_unexpectedEOF(self):

which permanently links to the appropriate version of the file, which means your line-number reference won't break the second someone updates the file 😄

[OliverFarren]

Not sure if this would be the relevant test for this issue. Not too familiar with the inner-workings of twisted. https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/test/test_tls.py#L1008

[glyph]

This behavior occurs every time a TLS connection is closed in such a manner that it raises ZeroReturnError without any arguments

Do you have any sense of what the "manner" in question here is?

[OliverFarren]

This behavior occurs every time a TLS connection is closed in such a manner that it raises ZeroReturnError without any arguments

Do you have any sense of what the "manner" in question here is?

I can provide a detailed stack-trace. Does this help clarify things?

  "body": {
    "trace_chain": [
      {
        "frames": [
          {
            "code": "why = method()",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/internet/asyncioreactor.py",
            "argspec": [
              "self",
              "selectable",
              "read"
            ],
            "lineno": 138,
            "method": "_readOrWrite",
            "locals": {
              "e": "<class 'IndexError'>",
              "read": true,
              "self": "<class 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'>",
              "method": "<class 'method'>",
              "selectable": "<class 'twisted.internet.tcp.Client'>",
              "why": "<class 'IndexError'>"
            }
          },
          {
            "code": "return self._dataReceived(data)",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/internet/tcp.py",
            "argspec": [
              "self"
            ],
            "lineno": 248,
            "method": "doRead",
            "locals": {
              "self": "<class 'twisted.internet.tcp.Client'>",
              "data": "b'\\x15\\x03\\x03\\x00\\x02\\x01\\x00'"
            }
          },
          {
            "code": "rval = self.protocol.dataReceived(data)",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/internet/tcp.py",
            "argspec": [
              "self",
              "data"
            ],
            "lineno": 253,
            "method": "_dataReceived",
            "locals": {
              "self": "<class 'twisted.internet.tcp.Client'>",
              "data": "b'\\x15\\x03\\x03\\x00\\x02\\x01\\x00'"
            }
          },
          {
            "code": "return self._wrappedProtocol.dataReceived(data)",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/internet/endpoints.py",
            "argspec": [
              "self",
              "data"
            ],
            "lineno": 151,
            "method": "dataReceived",
            "locals": {
              "self": "<class 'twisted.internet.endpoints._WrappingProtocol'>",
              "data": "b'\\x15\\x03\\x03\\x00\\x02\\x01\\x00'"
            }
          },
          {
            "code": "self._checkHandshakeStatus()",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/protocols/tls.py",
            "argspec": [
              "self",
              "bytes"
            ],
            "lineno": 324,
            "method": "dataReceived",
            "locals": {
              "self": "<class 'twisted.protocols.tls.BufferingTLSTransport'>",
              "bytes": "b'\\x15\\x03\\x03\\x00\\x02\\x01\\x00'"
            }
          },
          {
            "code": "self._tlsShutdownFinished(Failure())",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/protocols/tls.py",
            "argspec": [
              "self"
            ],
            "lineno": 251,
            "method": "_checkHandshakeStatus",
            "locals": {
              "self": "<class 'twisted.protocols.tls.BufferingTLSTransport'>"
            }
          },
          {
            "code": "if _representsEOF(reason.value):",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/protocols/tls.py",
            "argspec": [
              "self",
              "reason"
            ],
            "lineno": 377,
            "method": "_tlsShutdownFinished",
            "locals": {
              "self": "<class 'twisted.protocols.tls.BufferingTLSTransport'>",
              "reason": "<class 'twisted.python.failure.Failure'>"
            }
          },
          {
            "code": "errorQueue = exceptionObject.args[0]",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/protocols/tls.py",
            "argspec": [
              "exceptionObject"
            ],
            "lineno": 121,
            "method": "_representsEOF",
            "locals": {
              "exceptionObject": "<class 'OpenSSL.SSL.ZeroReturnError'>"
            }
          }
        ],
        "exception": {
          "message": "tuple index out of range",
          "class": "IndexError"
        }
      },
      {
        "frames": [
          {
            "code": "self._tlsConnection.do_handshake()",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/twisted/protocols/tls.py",
            "argspec": [
              "self"
            ],
            "lineno": 247,
            "method": "_checkHandshakeStatus",
            "locals": {
              "self": "<class 'twisted.protocols.tls.BufferingTLSTransport'>"
            }
          },
          {
            "code": "self._raise_ssl_error(self._ssl, result)",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/OpenSSL/SSL.py",
            "argspec": [
              "self"
            ],
            "lineno": 2193,
            "method": "do_handshake",
            "locals": {
              "self": "<class 'OpenSSL.SSL.Connection'>",
              "result": -1
            }
          },
          {
            "code": "raise ZeroReturnError()",
            "filename": "/usr/src/app/.venv/lib/python3.11/site-packages/OpenSSL/SSL.py",
            "argspec": [
              "self",
              "ssl",
              "result"
            ],
            "lineno": 1805,
            "method": "_raise_ssl_error",
            "locals": {
              "ssl": "<class '_cffi_backend.__CDataGCP'>",
              "self": "<class 'OpenSSL.SSL.Connection'>",
              "result": -1,
              "error": 6
            }
          }
        ],
        "exception": {
          "message": "",
          "class": "ZeroReturnError"
        }
      }
    ]
  }

[glyph]

I can provide a detailed stack-trace. Does this help clarify things?

Thanks, I don't think this is quite enough info, but it is at least a place to get started :)

[OliverFarren]

I can provide a detailed stack-trace. Does this help clarify things?

Thanks, I don't think this is quite enough info, but it is at least a place to get started :)

I assume i'm missing something, but isn't the issue that the _representsEOF method assumes that the incoming exception object was instantiated with args but that isn't always the case?

[glyph]

I assume i'm missing something, but isn't the issue that the _representsEOF method assumes that the incoming exception object was instantiated with args but that isn't always the case?

It is, but the question is why isn't that the case. I absolutely recognize the tendency here to say "oh, well, openssl is weird sometimes, let's just add more code to handle this bizarre new condition it threw at us" but this is a security-critical code path, and we should understand what exactly OpenSSL is trying to say (beyond "we are bad at software") by giving us an exception in this shape.

[OliverFarren]

I see, so the expectation is that an OpenSSL.SSL.ZeroReturnError would get invoked with arguments and this seems like a divergence from the norm?

On our side, this gets caught by our Error tracking system and occurs relatively rarely, perhaps 10 times over 14 days of our scrapy based application running in our production environment. Each time we get a stack trace similar to the one above. At the application level i'm not sure how I could reproduce. I can try running our service for a few days in a debug environment with some breakpoints set, that's not guaranteed to reproduce the set of conditions that reach this but i'd be happy to try. I'd need some guidance on where to set some breakpoints to inspect as i'm not too familiar with the inner workings of twisted or OpenSSL.

In anycase, let me know if there's anything I can do to help in understanding the issue more

[OliverFarren]

Looking at the current OpenSSL it seems like the ZeroReturnError is always raised without args?

[OliverFarren]

Perhaps a change to version of OpenSSL and/or cryptography has changed the flow?

Looking at our version history, in the last 6 months we've gone from using
pyopenssl == 23.3.0 to 24.1.0
changelog
And
cryptography == 41.0.5 to 42.0.7
changelog
And
twisted == 22.10.0 to 23.10.0

Our dependency on using 23.10.0 and not the newest is all down to scrapy's constraints.

[glyph]

Looking at the current OpenSSL it seems like the ZeroReturnError is always raised without args?

Yes, it always has been, and the one place in the code where Twisted explicitly handles a ZeroReturnError (i.e. here:

twisted/src/twisted/protocols/tls.py

Lines 289 to 295 in c465c46

	except ZeroReturnError:
	# TLS has shut down and no more TLS data will be received over
	# this connection.
	self._shutdownTLS()
	# Passing in None means the user protocol's connnectionLost
	# will get called with reason from underlying transport:
	self._tlsShutdownFinished(None)

) it doesn't expect it to have arguments. The question is, what operation is raising a ZeroReturnError elsewhere, and why is it doing that unexpectedly? Is the issue that we get it from do_handshake sometimes, rather than recv? I don't see the line numbers lining up here, but it seems likely that's what's going on. What does a ZeroReturnError from do_handshake, specifically, mean?

[OliverFarren]

Not hot on this stuff but could there have been a change to OpenSSL? Seems possible looking at some PR's

• openssl/openssl#21780
• openssl/openssl#21618
• https://github.com/search?q=repo%3Aopenssl%2Fopenssl+SSL_ERROR_ZERO_RETURN&type=pullrequests

[OliverFarren]

Perhaps also relevant in https://www.openssl.org/docs/man3.0/man7/migration_guide.html under TLS changes

New SSL option SSL_OP_IGNORE_UNEXPECTED_EOF

The SSL option SSL_OP_IGNORE_UNEXPECTED_EOF is introduced. If that option is set, an unexpected EOF is ignored, it pretends a close notify was received instead and so the returned error becomes SSL_ERROR_ZERO_RETURN.

[adiroiban]

I think that pyOpenSSL is fine.

ZeroReturnError is a very specific exception and has no extra text/details/message.... similar to WantRead or WantWrite.

I think that the issue is in Twisted code, as Twisted assumes that any error has an argument.

twisted/src/twisted/protocols/tls.py

Lines 113 to 123 in c465c46

	def _representsEOF(exceptionObject: Error) -> bool:
	"""
	Does the given OpenSSL.SSL.Error represent an end-of-file?
	"""
	reasonString: str
	if isinstance(exceptionObject, SysCallError):
	_, reasonString = exceptionObject.args
	else:
	errorQueue = exceptionObject.args[0]
	_, _, reasonString = errorQueue[-1]
	return reasonString.casefold().startswith("unexpected eof")

Just like we have special handling for SysCallError, we should also have special handling for other errors like ZeroReturnError

---

More info about the conditions for which this exception are raised can be found at https://www.openssl.org/docs/man3.3/man3/SSL_get_error.html

SSL_ERROR_ZERO_RETURN
The TLS/SSL peer has closed the connection for writing by sending the close_notify alert. No more data can be read. Note that SSL_ERROR_ZERO_RETURN does not necessarily indicate that the underlying transport has been closed.

This error can also appear when the option SSL_OP_IGNORE_UNEXPECTED_EOF is set. See SSL_CTX_set_options(3) for more details.

---

I don't know how easy is to replicate this error in an unit test

[OliverFarren]

I think that pyOpenSSL is fine.

ZeroReturnError is a very specific exception and has no extra text/details/message.... similar to WantRead or WantWrite.

I think that the issue is in Twisted code, assuming that any error has an argument

twisted/src/twisted/protocols/tls.py

Lines 113 to 123 in c465c46

	def _representsEOF(exceptionObject: Error) -> bool:
	"""
	Does the given OpenSSL.SSL.Error represent an end-of-file?
	"""
	reasonString: str
	if isinstance(exceptionObject, SysCallError):
	_, reasonString = exceptionObject.args
	else:
	errorQueue = exceptionObject.args[0]
	_, _, reasonString = errorQueue[-1]
	return reasonString.casefold().startswith("unexpected eof")

Just like we have special handling for SysCallError we should also have special handling for other errors like ZeroReturnError

More info about the conditions for which this exception are raised can be found at https://www.openssl.org/docs/man3.3/man3/SSL_get_error.html

SSL_ERROR_ZERO_RETURN
The TLS/SSL peer has closed the connection for writing by sending the close_notify alert. No more data can be read. Note that SSL_ERROR_ZERO_RETURN does not necessarily indicate that the underlying transport has been closed.
This error can also appear when the option SSL_OP_IGNORE_UNEXPECTED_EOF is set. See SSL_CTX_set_options(3) for more details.

I don't know how easy is to replicate this error in an unit test

I agree, I think it was more around understanding what's changed.

[OliverFarren]

@glyph, any more thoughts on this?

[glyph]

@glyph, any more thoughts on this?

It sounds like we have a handle on what's going on here, at least vaguely. I'd like to see a test for this; it should be possible to introduce an EOF by using a MemoryReactor style test. But if that's too much effort or we can't make it work where we think we should, a fake do_handshake wrapper that just directly tests we handle this case is fine.