Under certain conditions meta requests are retried after receiving non-recoverable S3 error responses. about aws-c-s3

[alexsmith1612]

I saw that conversation, and my reading of the RFC is that no additional requests following the current in-progress request may be sent after receiving Connection: close. The RFC 9112 Section 9.5 paragraph 5 that Michael quoted in particular seems to heavily imply to me that the in-progress body does not need to be immediately terminated when a non-error response with Connection: close is received.

Looking at curl's commit history, their behavior of terminating of in-progress body transmissions upon receipt of a 200 response with Connection: close (or the absence of Connection: keep-alive for HTTP/1.0) started with curl/curl@c67a99f. Following the bug report in that commit message, it appears that this was implemented due to curl continuing to send data after receiving error responses with Connection: close, with no discussion about non-error responses. Based on that, it seems to me that the curl behavior on a 200 response with Connection: close may have been less of an explicit design choice and more of an unhandled corner case.

That being said, since this curl behavior has existed since 2008, that probably means that this is not a situation that is encountered often, so it may not matter what the client's behavior is in practice.

[TingDaoK]

Thank you for reporting the issue and the detail investigation!
We just merged the fix from aws-c-http, it should fix the issue here.

[graebm]

Which language CRT are you using to get at aws-c-s3? aws-crt-java? aws-crt-cpp?

btw thank you for the AMAZING deep dive writeup. It really helped us dive right into a fix

[alexsmith1612]

Thank you for reporting the issue and the detail investigation!
We just merged the fix from aws-c-http, it should fix the issue here.

Awesome! I'll give it a test soon to ensure that my issue has been fixed.

Which language CRT are you using to get at aws-c-s3? aws-crt-java? aws-crt-cpp?

I'm using the C CRT libraries directly.

btw thank you for the AMAZING deep dive writeup. It really helped us dive right into a fix

Of course! I try to provide the level of detail in problem descriptions and debug information that I'd like to receive myself. Thank you two for the quick turnaround.

[alexsmith1612]

The changes in awslabs/aws-c-http@0600662 appear to fix the issue, thanks!

FWIW, I haven't encountered this as an issue, but it does seem to me that it might make sense to finish transmitting the (current) request even after receiving a complete Connection: close response if the response status is < 300, although I'm not sure how often that situation arises in practice. If such a change did get implemented, I believe that an additional change would be needed to ensure that reads (on leftward slots) do truly stop after the client finishes processing the response while transmission continues. But I digress.

[TingDaoK]

We thought about it. part of conversation can be found here

The reasons that we just end transmitting after connection: close no matter what response status:

• The spec said MUST cease, it's a bit ambiguous

A client that receives a "close" connection option MUST cease sending requests on that connection and close the connection after reading the response message containing the "close" connection option

• I also tested with curl against a local host. curl also stops sending with 200 and connection: close, some log from curl

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< Server: mock_s3_server
< ETag: b54357faf0632cce46e942fa68356b38
< Connection: close
< Content-Length: 0
< 
* we are done reading and this is set to close, stop send

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< Server: mock_s3_server
< ETag: b54357faf0632cce46e942fa68356b38
< Connection: keep-alive
< Content-Length: 0
< 
* We are completely uploaded and fine