schannel_recv treats no bytes available as end-of-stream about curl

[chris-araman]

Regressed here:
145c263

[mback2k]

Thanks, merge request #245 just landed in aa99a63.

[jay]

Re sourceforge bug 1462 it basically was schannel_recv returning the count of encrypted bytes returned by raw recv (via Curl_read_plain) rather than the count of decrypted bytes when encrypted bytes were received but no decrypted data was available (either pre or post processed from the cache and/or that just received data). schannel_recv should never return the count of encrypted bytes so that was definitely a bug and was solved by returning 0 in the case no data was decrypted. And that causes this issue.

@chris-araman it looks like you have already figured that out but for anyone else who's interested that's the background.

@mback2k regarding the recent commits related to this issue:

If you go with Chris' solution then when a raw recv fails schannel_recv returns immediately rather than continuing to process any already cached encrypted and decrypted data. That doesn't look right to me, shouldn't it continue on and handle the caches as it did before? For performance even if not required.

Also is it possible for schannel_recv to still return 0 when no decrypted data is available or !len. The former would be a problem, the latter I don't know and it isn't documented for curl_easy_recv what should happen when buflen is 0.

How about:

• return 0 if len == 0. that will have to be documented.
• continue on and process the caches regardless of raw recv
• if decrypted data will be returned then set the error code to CURLE_OK and return its count
• if decrypted data will not be returned and the connection has closed (eg nread == 0) then return 0 and CURLE_OK
• if decrypted data will not be returned and the connection hasn't closed then set the error code to CURLE_AGAIN --only if an error code isn't already set-- and return -1
• narrow the Win2k workaround to only Win2k

A draft is here.

[jay]

Reopening this due to problems with aa99a63 as described above and since confirmed. Please do not use that fix in production, we're still working on this issue.

[mback2k]

@jay Did you see my inline comments about your proposed solution? Thanks for that, btw.

[jay]

@mback2k I did, sorry I left you hanging. I started a second draft and wanted to post it before I got back to you. So far it is similar to the first. I'll probably have it up within a week.

[jay]

I just finished draft 4, the branch is here if anyone wants to try it out.

[jay]

@bagder I'm pretty much near the end of my work on this but I don't have a review yet. I'd like to get this into 7.43 because as it stands now we have problems with schannel. What I have in the branch covers more corner cases than what's in master now.

It's likely further changes will be needed to address that data-always-pending issue but I cannot reproduce ..yet.

Still I am down to the wire here, please let me know what you think I should do.

[bagder]

@mback2k is really the most clueful person to get input from on schannel details. I can only like check for code style violations and I know you're very good at not doing such...

So it is really a question of confidence level. I will package the release in ~8 hours so if you truly believe that this patch improves the situation then merge it. If you have doubts I think it is better to be safe than sorry. We could even consider a 7.43.1 release later if we feel a need to get some more bug fixes out before 7.44.0 time. I trust your judgement here!

[jay]

I will package the release in ~8 hours so if you truly believe that this patch improves the situation then merge it.

Landed in 3e7ec1e.

[chris-araman]

I'm testing 7.43.0 out now. Looks good so far. Thanks, @jay and @mback2k!

[mback2k]

@jay Thank you very much for your time and work on this. I have not yet been able to test it myself, but my automatic builds seem fine, e.g.:
https://curlbuild.uxnr.de/builders/curl_winssl_mingw_x86/builds/1714
https://curlbuild.uxnr.de/builders/curl_winssl_mingw_w64_x86/builds/1593
https://curlbuild.uxnr.de/builders/curl_winssl_mingw_w64_x64/builds/2092

[jay]

No problem guys. I'm closing this but Marc if you have some time before the next release and you are able to do a review that would be great.

[mback2k]

@jay Do you think there is an easy way to test the different scenarios schannel_recv has to handle, including re-negotiation? I mean, my autobuilds and personal tests look fine, but schannel_recv was and still is quite complex.

[mback2k]

@jay and @bagder: All in all the current (new) schannel_recv looks a lot better and more stable regarding error handling than my original one. Thank you very much, @jay and everyone who helped.

I am just thinking about the removed Windows 2000 specific alert check. The code still checks for Windows 2000 in https://github.com/bagder/curl/blob/master/lib/vtls/schannel.c#L1113-L1163, but it just assumes that the connection was closed graceful instead of checking for a SSL/TLS alert packet.

I guess the check was removed because the SSL/TLS alert packet could also include data which is now processed (decrypted) and therefore no longer available if not cached for this purpose.

[jay]

Marc, Thanks for your positive comments. Unfortunately I am not aware of an easy automated way to test schannel_recv. I tested it manually as I was fixing it but I don't have specific tests in the suite for each change that was made.

Renegotiation: I did not test that. It was unmodified save for the fact it no longer can cause recv recursion and I banned renegotiation when encrypted data was received after the renegotiation message but before libcurl starts it. The former was for security, the latter I got from Chromium or Microsoft's SDK (or a lack of handling that scenario in their code). The code and path for renegotiation once it leaves schannel_recv is unmodified (by me, anyway). I am unaware of a renegotiation test in the suite. Is there an easy way to test renegotiation? I will probably not get around to it myself but I think it would be a good test to have.

Win2k alert: We discussed it earlier this year in draft comments. My sentiment is the same but in addition here is more. To frame this, a connection was closed and then there was a magic byte check 0x15 in the encrypted data. My understanding of this is you were trying to detect whether or not an encrypted TLS alert has taken place (0x15) that wasn't decrypted. I don't see how we could tell from checking the encrypted data whether we've got a valid record that contains a close_notify. It's on schannel to do that, though it doesn't. So in Win2k I think it makes more sense to just assume we have the close_notify if it doesn't let us know for sure. No missing close_notify means no error, though those clients lose protection from truncation attacks.

[mback2k]

@jay You could use my own test URL for that. It works by specifying different (in this case less secure) ciphers for the directory:

        <Location /renegotiate/>
                SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+EXP
        </Location>

And it seems to work great:

$ ./src/curl.exe -v https://stuff.marc-hoersken.de/renegotiate/
* STATE: INIT => CONNECT handle 0x31ef00; line 1103 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* timeout on name lookup is not supported
*   Trying 2a00:1828:2000:378:1:80:59ee:542e...
* STATE: CONNECT => WAITCONNECT handle 0x31ef00; line 1156 (connection #0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to stuff.marc-hoersken.de (2a00:1828:2000:378                                                                                                                                        :1:80:59ee:542e) port 443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x31ef00; line 1253 (connection #0)
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 191 bytes...
* schannel: sent initial handshake data: sent 191 bytes
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x31ef00; line 1267 (connection #0)
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 1440 length 4096
* schannel: encrypted data length: 1338
* schannel: encrypted data buffer: offset 1338 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 3994 length 4096
* schannel: encrypted data length: 132
* schannel: encrypted data buffer: offset 132 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 2308 length 4096
* schannel: sending next handshake data: sending 182 bytes...
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 107 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 3/3)
* schannel: incremented credential handle refcount = 1
* schannel: stored credential handle in session cache
* STATE: PROTOCONNECT => DO handle 0x31ef00; line 1288 (connection #0)
> GET /renegotiate/ HTTP/1.1
> Host: stuff.marc-hoersken.de
> User-Agent: curl/7.47.0-DEV
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x31ef00; line 1350 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x31ef00; line 1477 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x31ef00; line 1487 (connection #0)
* schannel: client wants to read 16384 bytes
* schannel: encdata_buffer resized 17408
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 85
* schannel: encrypted data buffer: offset 85 length 17408
* schannel: decrypted data length: 0
* schannel: decrypted data added: 0
* schannel: decrypted data cached: offset 0 length 16384
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: sending next handshake data: sending 277 bytes...
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 1440 length 17408
* schannel: encrypted data length: 1243
* schannel: encrypted data buffer: offset 1243 length 17408
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 2683 length 17408
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 3899 length 17408
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 5339 length 17408
* schannel: encrypted data length: 1398
* schannel: encrypted data buffer: offset 1398 length 17408
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 2527 length 17408
* schannel: sending next handshake data: sending 335 bytes...
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 2/3)
* schannel: encrypted data buffer: offset 186 length 17408
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with stuff.marc-hoersken.de port 443 (step 3/3)
* schannel: incremented credential handle refcount = 2
* schannel: SSL/TLS connection renegotiated
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: decrypted data buffer: offset 0 length 16384
* schannel: schannel_recv cleanup
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 650
* schannel: encrypted data buffer: offset 650 length 17408
* schannel: decrypted data length: 194
* schannel: decrypted data added: 194
* schannel: decrypted data cached: offset 194 length 16384
* schannel: encrypted data length: 373
* schannel: encrypted data cached: offset 373 length 17408
* schannel: decrypted data length: 299
* schannel: decrypted data added: 299
* schannel: decrypted data cached: offset 493 length 16384
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: decrypted data buffer: offset 493 length 16384
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 493
* schannel: decrypted data buffer: offset 0 length 16384
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 404 Not Found
< Date: Wed, 23 Dec 2015 11:20:37 GMT
* Server Apache/2.4.10 (Debian) is not blacklisted
< Server: Apache/2.4.10 (Debian)
< Content-Length: 299
< Content-Type: text/html; charset=iso-8859-1
< Via: 1.1 stuff.marc-hoersken.de
<
{ [299 bytes data]
* STATE: PERFORM => DONE handle 0x31ef00; line 1645 (connection #0)
* Curl_done
100   299  100   299    0     0   1196      0 --:--:-- --:--:-- --:--:--  1196<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /renegotiate/ was not found on this server.</p>
<hr>
<address>Apache/2.4.10 (Debian) Server at stuff.marc-hoersken.de Port 443</address>
</body></html>

* Connection #0 to host stuff.marc-hoersken.de left intact

I am not sure if renegotation is testable using stunnel, because that does care about the actual content (in this case the HTTP URL) being transferred.