HTTP connection closed on: Excess found in a non pipelined read about curl

[michael-o]

Here is another log of

curl --negotiate -u : -T "file.pdf" http://.../[1-2].pdf -o /dev/null --verbose -s

*   Trying <server-ip>...
* Connected to <server-hostname> (<server-ip>) port 8081 (#0)
> PUT /webapp/files/1.pdf HTTP/1.1
> Host: <server-hostname>:8081
> User-Agent: curl/7.42.0
> Accept: */*
> Content-Length: 20480
> Expect: 100-continue
> 
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Length: 974
< Date: Sun, 26 Apr 2015 20:24:01 GMT
< 
* Excess found in a non pipelined read: excess = 974 url = /webapp/files/1.pdf (zero-length body)
* Closing connection 0
* Issue another request to this URL: 'http://<server-hostname>:8081/webapp/files/1.pdf'
* Hostname <server-hostname> was found in DNS cache
*   Trying <server-ip>...
* Connected to <server-hostname> (<server-ip>) port 8081 (#1)
* Server auth using Negotiate with user ''
> PUT /webapp/files/1.pdf HTTP/1.1
> Host: <server-hostname>:8081
> Authorization: Negotiate YIIP7wYGKwYBBQU...T
> User-Agent: curl/7.42.0
> Accept: */*
> Content-Length: 20480
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
} [16384 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 405 Method Not Allowed
< Server: Apache-Coyote/1.1
< WWW-Authenticate: Negotiate oYHtMIHqo...
< Connection: close
< Allow: GET,OPTIONS
< Content-Type: text/html;charset=utf-8
< Content-Length: 1088
< Date: Sun, 26 Apr 2015 20:24:01 GMT
< 
{ [1088 bytes data]
* Closing connection 1
* Hostname <server-hostname> was found in DNS cache
*   Trying <server-ip>...
* Connected to <server-hostname> (<server-ip>) port 8081 (#2)
> PUT /webapp/files/2.pdf HTTP/1.1
> Host: <server-hostname>:8081
> User-Agent: curl/7.42.0
> Accept: */*
> Content-Length: 20480
> Expect: 100-continue
> 
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Length: 974
< Date: Sun, 26 Apr 2015 20:24:01 GMT
< 
* Excess found in a non pipelined read: excess = 974 url = /webapp/files/2.pdf (zero-length body)
* Closing connection 2
* Issue another request to this URL: 'http://<server-hostname>:8081/webapp/files/2.pdf'
* Hostname <server-hostname> was found in DNS cache
*   Trying <server-ip>...
* Connected to <server-hostname> (<server-ip>) port 8081 (#3)
* Server auth using Negotiate with user ''
> PUT /webapp/files/2.pdf HTTP/1.1
> Host: <server-hostname>:8081
> Authorization: Negotiate YIIP7wYGKwYB...
> User-Agent: curl/7.42.0
> Accept: */*
> Content-Length: 20480
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
} [16384 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 405 Method Not Allowed
< Server: Apache-Coyote/1.1
< WWW-Authenticate: Negotiate oYHtMIHqoAMK...
< Connection: close
< Allow: GET,OPTIONS
< Content-Type: text/html;charset=utf-8
< Content-Length: 1088
< Date: Sun, 26 Apr 2015 20:24:01 GMT
< 
{ [1088 bytes data]
* Closing connection 3

Please take a closer look at the Content-Length sent by the server: 974. This is the same number reported by excess read and telling that body is zero length. I would have assumed that ths body should be fully consumed silently. Ist the request body read at all?

[bagder]

Curl_http_readwrite_headers() (https://github.com/bagder/curl/blob/master/lib/http.c#L2903) is the function that returns stop_reading and that ends up in that error message.

As you can see, there are only two reasons stop_reading can be returned true in that function. Can you figure out which triggers in your case?

It is called from https://github.com/bagder/curl/blob/master/lib/transfer.c#L480

[iboukris]

Hi,

I think I can reproduce this (or at least similar) I get:

* Marked for [closure]: Mid-auth HTTP and much data left to send
<
* Excess found in a non pipelined read: excess = 1656 url = /mika/little_upfile (zero-length body)
* Closing connection 0

Looks like it might be by some mysterious design, see:
https://github.com/bagder/curl/blob/master/lib/http.c#L458
https://github.com/bagder/curl/blob/master/lib/README.httpauth

I get the same when I set CURLOPT_HTTPAUTH to (CURLAUTH_BASIC | CURLAUTH_ONLY) as it starts with an empty request...

[bagder]

Can you give me a full recipe on how to repeat this myself?

[bagder]

And it isn't that mysterious really, just a matter of trying to decide when it makes sense to close things for the next request or when to keep the connection alive and wastefully sending data.

[michael-o]

@bagder, I checked both spots and one refers to download which should not apply here. The other one checks whether there is a request body. That is confusing because I do provide one. Reading README.httpauth, first point. This makes vague assumptions. I would expect a concrete approach: POST/PUT, Expect, 4xx/5xx, continue or fail.

[bagder]

I don't understand your point. What "concrete approach" are you missing?

The problem with Negotiate not being treated connection-oriented is still there btw and contributes to this closure.

[michael-o]

I am consufed about:

If a 401 (or 407 when talking through a proxy) is received, then:

If we have "more than just a little" data left to send, close the
connection. Exactly what "more than just a little" means will have to be
determined. Possibly the current transfer speed should be taken into
account as well.

Especially about "just a little".

Deducing from your answer in #223, I assumbed the connection was closed due to the read excess and not the Negotiate auth.

[iboukris]

From the example section - note the CURLOPT_HTTPAUTH if you set to CURLAUTH_BASIC only it will not occur (it will if you set to CURLAUTH_GSSNEGOTIATE only).

#include <stdio.h>
#include <string.h>
#include <curl/curl.h>

const char data[]="this is what we post to the silly web server";

struct WriteThis {
  const char *readptr;
  long sizeleft;
};

static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *userp)
{
  struct WriteThis *pooh = (struct WriteThis *)userp;

  if(size*nmemb < 1)
    return 0;

  if(pooh->sizeleft) {
    *(char *)ptr = pooh->readptr[0]; /* copy one single byte */
    pooh->readptr++;                 /* advance pointer */
    pooh->sizeleft--;                /* less data left */
    return 1;                        /* we return 1 byte at a time! */
  }

  return 0;                          /* no more data left to deliver */
}

int main(void)
{
  CURL *curl;
  CURLcode res;

  struct WriteThis pooh;

  pooh.readptr = data;
  pooh.sizeleft = (long)strlen(data);

  /* In windows, this will init the winsock stuff */
  res = curl_global_init(CURL_GLOBAL_DEFAULT);
  /* Check for errors */
  if(res != CURLE_OK) {
    fprintf(stderr, "curl_global_init() failed: %s\n",
            curl_easy_strerror(res));
    return 1;
  }

  curl = curl_easy_init();
  if(curl) {
    curl_easy_setopt(curl, CURLOPT_POST, 1L);

    curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);

    curl_easy_setopt(curl, CURLOPT_READDATA, &pooh);

    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

    curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, pooh.sizeleft);
    curl_easy_setopt(curl, CURLOPT_URL, "http://ms.frenche.cp/mika/");
    curl_easy_setopt(curl, CURLOPT_USERPWD, "usera:passa");
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, (long)CURLAUTH_BASIC | CURLAUTH_ONLY);
#ifdef DISABLE_EXPECT
    {
      struct curl_slist *chunk = NULL;

      chunk = curl_slist_append(chunk, "Expect:");
      res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
    }
#endif

    res = curl_easy_perform(curl);
    if(res != CURLE_OK)
      fprintf(stderr, "curl_easy_perform() failed: %s\n",
              curl_easy_strerror(res));

    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
  return 0;
}

[michael-o]

@Frenche, please note that I did not disable the Expect header.

[iboukris]

@michael-o me neither, it is just from the example.
Compiling with '-DDISABLE_EXPECT' does not make much difference.
Same goes when i set 'CURLOPT_UPLOAD' to 1 (only then my server says 501 Not Implemented).
Also note it seem to use 'Transfer-Encoding: chunked', I can paste full outputs...

Strangely, sometimes the second request fails and I get:

* necessary data rewind wasn't possible
* Marked for [closure]: Transfer returned error
* Closing connection 0
* The cache now contains 0 members
* Expire cleared
curl_easy_perform() failed: Send failed since rewinding of the data stream failed

[michael-o]

@Frenche, if you are interested, I could run 10 uploads with curl --next with the same connection if curl allows. We'll se how curl reacts.

[iboukris]

On Sun, Apr 26, 2015 at 9:31 PM, Daniel Stenberg [email protected]
wrote:

The problem with Negotiate not being treated connection-oriented is still
there btw and contributes to this closure.

I'm not sure, in Michael's case it could very well be treated as
request-based since he uses kerberos (and i thin should).
Currently the code seem to considered it request-based since
state.authhost.done is set to true even when gssapi returns 'continue'.

It seem more related to the bug that Negotiate code sends an empty request
unnecessarily (a common mechanism to send an initial empty request already
exist).
I can try to start with a better version of my original patch for this
(while preparing background for more suggestions).
That should solve the problem when using --negotiate but it won't solve the
possible main issue that will still occur with any 'set' of auth (like
--anyauth).

Regards,
Isaac B.

[michael-o]

Currently the code seem to considered it request-based since
state.authhost.done is set to true even when gssapi returns 'continue'.

That does not sound right to me..

It seem more related to the bug that Negotiate code sends an empty request
unnecessarily (a common mechanism to send an initial empty request already
exist).

Isn't this premature optimization? I hardly believe this is requested by the user. If this is really necessary, the client should do this on his own. I wouldn't the library to do that unless I request it to do.

[iboukris]

On Sun, Apr 26, 2015 at 11:26 PM, Michael Osipov [email protected]
wrote:

Currently the code seem to considered it request-based since
state.authhost.done is set to true even when gssapi returns 'continue'.

That does not sound right. This is false security to me.

I don't see any security issue caused by that.

It seem more related to the bug that Negotiate code sends an empty
request
unnecessarily (a common mechanism to send an initial empty request already
exist).

Isn' this premature optimization? I hardly believe this is requested by
the user. If this is really necessary, the client should do this on his
own. I wouldn't the library to do that unless I request it to do.

In my opinion this is the correct thing to do, the user can always get the
initial empty request by ORing with CURLAUTH_ONLY as mentioned or by using
--anyauth which is the same for Negotiate since it will get picked if
supported ;-)

It will also align negotiate with the other auth protocols, but mainly it
will allow to send without the empty request as allowed by the RFC.

[bagder]

Honestly, I see no point in chasing this problem until we have Negotiate fixed to be connection-oriented like NTLM, as that will change details such as this anyway.

[iboukris]

@bagder note that in the example I mentioned there is no Negotiate I think any 'set' will result the same closure of connection which seem a little strange.

[bagder]

Sorry I drowned in all the different directions this issue is going. Can you give me a recipe (again?) for such an excess output without Negotiate - and one that can be considered a problem ?

[michael-o]

On Sun, Apr 26, 2015 at 11:26 PM, Michael Osipov [email protected] wrote:
Currently the code seem to considered it request-based since
state.authhost.done is set to true even when gssapi returns 'continue'.

That does not sound right. This is false security to me.

I don't see any security issue caused by that.

We probably misunderstood each other. If you say that the libcurl assumes that the auth is done while the GSS context says continue, this is a security issue because mutual authentication has not been completed and the target server is not trustable.

I agree on the second.

@bagder,

Honestly, I see no point in chasing this problem until we have Negotiate fixed to be connection-oriented like NTLM, as that will change details such as this anyway.

This is not always true as I have said already. If Kerberos is selected it can be request-based. gss_init_sec_context provides an output parameter &actual_mech which tells you the actual mech. SSPI offer something similar. This should probably evaluated.

[bagder]

@michael-o I know that, but as it can be connection-oriented and I don't know if we can tell the difference, we break badly right now when it is connection-oriented and we need to fix that. It requires a big take on the problem and that will change subtle details as this.

[iboukris]

@bagder - sorry about that - please see:
#232 (comment)

[iboukris]

On Mon, Apr 27, 2015 at 11:40 AM, Michael Osipov [email protected]
wrote:

On Sun, Apr 26, 2015 at 11:26 PM, Michael Osipov [email protected]
wrote:
Currently the code seem to considered it request-based since
state.authhost.done is set to true even when gssapi returns 'continue'.

That does not sound right. This is false security to me.
I don't see any security issue caused by that.

We probably missunderstood each other. If you say that the libcurl
assumes that the auth is done while the GSS context says continue, this is
a security issue because mutual authentication has not been completed and
the target server is not trustable.

It assumes auth done for some upload related decisions etc, I didn't notice
an impact on security derived from this.

We actually never check mutual-auth... in fact we never fully establish the
context because we use SPNEGO hard-coded (we should definitely let the app
chose) where the last token always come from server to client in the last
200 OK packet which we don't check at all for www-authenticate.

[michael-o]

@Frenche,

that is a serious flaw and must be fixed immediately. Additionally, users should know about that flaw too. As long as this is not fixed, I would highly recommend remove the MUTUAL_AUTH flag and add a to do. Surprisingly, I have fould the very same problems in the Apache HttpClient where I took over the leadership to overhaul GSS-API auth (discussion).

Unfortunately, I am not much of a C hacker to provide valueable patches but I can at least review code, provide concepts and test infrastructure.

[iboukris]

On Mon, Apr 27, 2015 at 12:17 PM, Michael Osipov [email protected]
wrote:

@Frenche https://github.com/frenche,

that is a serious flaw and must be fixed immediately. Additionally, users
should know about that flaw too. As long as this is not fixed, I would
highly recommend remove the MUTUAL_AUTH flag and add a to do.
Surprisingly, I have fould the very same problems in the Apache HttpClient
where I took over the leadership to overhaul GSS-API auth
https://issues.apache.org/jira/browse/HTTPCLIENT-1625.

And I found similar issue in 'mod_auth_gssapi' ;-)
See: https://github.com/modauthgssapi/mod_auth_gssapi/issues/21

But frankly, I don't think this is much of a security either, see long
discussion about it:
https://bugzilla.mozilla.org/show_bug.cgi?id=17578

What it means is that despite Negotiate being such a 'wonderful' protocol,
implementations don't always get it right so one need to consider that as
well...

[michael-o]

@Frenche, please read comments 242 to 244 in the Mozilla issue. They have decided to disable mutual auth. This is the right thing to do if you cannot do it right.

The problem is not SPNEGO. Generally, GSS-API mechanisms are intended to be complete before client communication starts. This works perfectly with SASL. Unfortunately, HTTP is crappy here. It is not connection-oriented and mixes authentication with client communication. This is our (serious) problem, not the mechanism. More over, the response token can be on any status, not only 200. It can be 2xx, 3xx or 4xx.

The simple answer why people don't get it right is because they don't read the RFCs and simply do not comprehend how context establishment is support to work. I see this too often on Stack Overflow and other sites. (I agree that this is hard stuff)

Why it matters to me? If curl aims to be best-in-class (which it is to me), I'd expect to have best-in-class auth support. I/we gained so much from curl that I feel obliged to give much back.

[michael-o]

I have compiled a modified version of curl without mutual authentication and ran it against two servers:

Against Apache Tomcat:

~/curl$ bin/curl --verbose --negotiate -u : -o /dev/null http://<hostname>:8081/manager/html -s
*   Trying <server_ip>...
* Connected to <hostname> (<server_ip>) port 8081 (#0)
> GET /manager/html HTTP/1.1
> Host: <hostname>:8081
> User-Agent: curl/7.42.0-DEV
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 01:00:00 CET
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Length: 974
< Date: Mon, 27 Apr 2015 12:32:51 GMT
<
* Ignoring the response-body
{ [974 bytes data]
* Connection #0 to host <hostname> left intact
* Issue another request to this URL: 'http://<hostname>:8081/manager/html'
* Found bundle for host <hostname>: 0x28828c50
* Re-using existing connection! (#0) with host <hostname>
* Connected to <hostname> (<server_ip>) port 8081 (#0)
* Server auth using Negotiate with user ''
> GET /manager/html HTTP/1.1
> Host: <hostname>:8081
> Authorization: Negotiate YIIPPAYGKwYBBQUCoIIPMDCCDyygJz...
> User-Agent: curl/7.42.0-DEV
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 01:00:00 CET
< WWW-Authenticate: Negotiate oRQwEqADCgEAoQsGCSqGSIb3EgECAg==
< Connection: close
< Set-Cookie: JSESSIONID=034F68214883203E9BB26422CBA8444F; Path=/manager; HttpOnly
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Vary: Accept-Encoding
< Date: Mon, 27 Apr 2015 12:32:51 GMT
<
{ [3944 bytes data]
* Closing connection 0

Against Apache Web Server:

~/curl$ bin/curl --verbose --negotiate -u : -o /dev/null -k -s https://<hostname>/repos/websvn/
*   Trying <server_ip>...
* Connected to <hostname> (<server_ip>) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2, TLS Unknown, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, Client hello (1):
} [512 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, Server hello (2):
{ [98 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, CERT (11):
{ [1873 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, Server key exchange (12):
{ [589 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, Server finished (14):
{ [4 bytes data]
* SSLv2, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, Client key exchange (16):
} [70 bytes data]
* SSLv2, Unknown (20):
} [5 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
} [1 bytes data]
* SSLv2, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
} [16 bytes data]
* SSLv2, Unknown (20):
{ [5 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
{ [1 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*        subject: ...
*        start date: 2014-06-24 08:00:33 GMT
*        expire date: 2015-06-24 08:00:33 GMT
*        issuer: ...
*        SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* SSLv2, Unknown (23):
} [5 bytes data]
> GET /repos/websvn/ HTTP/1.1
> Host: <hostname>
> User-Agent: curl/7.42.0-DEV
> Accept: */*
>
* SSLv2, Unknown (23):
{ [5 bytes data]
< HTTP/1.1 401 Authorization Required
< Date: Mon, 27 Apr 2015 12:34:58 GMT
< Server: Apache/2.2.29 (FreeBSD) SVN/1.8.13 PHP/5.4.39 mod_ssl/2.2.29 OpenSSL/1.0.2a DAV/2
< WWW-Authenticate: Negotiate
< Content-Length: 553
< Content-Type: text/html; charset=iso-8859-1
<
* SSLv2, Unknown (23):
{ [5 bytes data]
* Ignoring the response-body
{ [553 bytes data]
* Connection #0 to host <hostname> left intact
* Issue another request to this URL: 'https://<hostname>/repos/websvn/'
* Found bundle for host <hostname>: 0x28828c50
* Re-using existing connection! (#0) with host <hostname>
* Connected to <hostname> (<server_ip>) port 443 (#0)
* Server auth using Negotiate with user ''
* SSLv2, Unknown (23):
} [5 bytes data]
> GET /repos/websvn/ HTTP/1.1
> Host: <hostname>
> Authorization: Negotiate YIIPPQYGKwYBBQUCoIIPMTCCDy2gJzAlBgkqhkiG...
> User-Agent: curl/7.42.0-DEV
> Accept: */*
>
* SSLv2, Unknown (23):
{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Mon, 27 Apr 2015 12:34:58 GMT
< Server: Apache/2.2.29 (FreeBSD) SVN/1.8.13 PHP/5.4.39 mod_ssl/2.2.29 OpenSSL/1.0.2a DAV/2
< WWW-Authenticate: Negotiate oRQwEqADCgEAoQsGCSqGSIb3EgECAg==
< X-Powered-By: PHP/5.4.39
< Content-Language: de
< Content-Length: 6155
< Content-Type: text/html; charset=UTF-8
<
* SSLv2, Unknown (23):
{ [5 bytes data]
* Closing connection 0
* SSLv2, Unknown (21):
} [5 bytes data]
* TLSv1.2, TLS alert, Client hello (1):
} [2 bytes data]

Both JGSS (Apache Tomcat) and MIT Kerberos (Apache Web Server) still respond with a token. So, either way one needs to pass the token back.

[iboukris]

That's because - as I mentioned - we currenly use SPNEGO hard-coded so
regardless if you request mutual-auth you still need an answer from sever
to confirm SPNEGO negotiation finished or still going.

If you give the address of 'Curl_krb5_mech_oid' to 'gss_init_sec_context'
you will see that it returns COMPLETE immediately not CONTINUE and you'll
see server response is empty.

On Mon, Apr 27, 2015 at 3:40 PM, Michael Osipov [email protected]
wrote:

I have compiled a modified version of curl without mutual
authentication and ran it against two servers:

~/curl$ bin/curl --verbose --negotiate -u : -o /dev/null http://:8081/manager/html -s

• Trying <server_ip>...
• Connected to (<server_ip>) port 8081 (#0)

  GET /manager/html HTTP/1.1
  Host: :8081
  User-Agent: curl/7.42.0-DEV
  Accept: /

  < HTTP/1.1 401 Unauthorized
  < Server: Apache-Coyote/1.1
  < Cache-Control: private
  < Expires: Thu, 01 Jan 1970 01:00:00 CET
  < WWW-Authenticate: Negotiate
  < Content-Type: text/html;charset=utf-8
  < Content-Length: 974
  < Date: Mon, 27 Apr 2015 12:32:51 GMT
  <
• Ignoring the response-body
  { [974 bytes data]
• Connection #0 to host left intact
• Issue another request to this URL: 'http://:8081/manager/html'
• Found bundle for host : 0x28828c50
• Re-using existing connection! (#0) with host
• Connected to (<server_ip>) port 8081 (#0)
• Server auth using Negotiate with user ''

  GET /manager/html HTTP/1.1
  Host: :8081
  Authorization: Negotiate YIIPPAYGKwYBBQUCoIIPMDCCDyygJz...
  User-Agent: curl/7.42.0-DEV
  Accept: /

  < HTTP/1.1 200 OK
  < Server: Apache-Coyote/1.1
  < Cache-Control: private
  < Expires: Thu, 01 Jan 1970 01:00:00 CET
  < WWW-Authenticate: Negotiate oRQwEqADCgEAoQsGCSqGSIb3EgECAg==
  < Connection: close
  < Set-Cookie: JSESSIONID=034F68214883203E9BB26422CBA8444F; Path=/manager; HttpOnly
  < Content-Type: text/html;charset=utf-8
  < Transfer-Encoding: chunked
  < Vary: Accept-Encoding
  < Date: Mon, 27 Apr 2015 12:32:51 GMT
  <
  { [3944 bytes data]
• Closing connection 0

~/curl$ bin/curl --verbose --negotiate -u : -o /dev/null -k -s https:///repos/websvn/

• Trying <server_ip>...
• Connected to (<server_ip>) port 443 (#0)
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@strength
• successfully set certificate verify locations:
• CAfile: /etc/ssl/cert.pem
  CApath: none
• TLSv1.2, TLS Unknown, Unknown (22):
  } [5 bytes data]
• TLSv1.2, TLS handshake, Client hello (1):
  } [512 bytes data]
• SSLv2, Unknown (22):
  { [5 bytes data]
• TLSv1.2, TLS handshake, Server hello (2):
  { [98 bytes data]
• SSLv2, Unknown (22):
  { [5 bytes data]
• TLSv1.2, TLS handshake, CERT (11):
  { [1873 bytes data]
• SSLv2, Unknown (22):
  { [5 bytes data]
• TLSv1.2, TLS handshake, Server key exchange (12):
  { [589 bytes data]
• SSLv2, Unknown (22):
  { [5 bytes data]
• TLSv1.2, TLS handshake, Server finished (14):
  { [4 bytes data]
• SSLv2, Unknown (22):
  } [5 bytes data]
• TLSv1.2, TLS handshake, Client key exchange (16):
  } [70 bytes data]
• SSLv2, Unknown (20):
  } [5 bytes data]
• TLSv1.2, TLS change cipher, Client hello (1):
  } [1 bytes data]
• SSLv2, Unknown (22):
  } [5 bytes data]
• TLSv1.2, TLS handshake, Finished (20):
  } [16 bytes data]
• SSLv2, Unknown (20):
  { [5 bytes data]
• TLSv1.2, TLS change cipher, Client hello (1):
  { [1 bytes data]
• SSLv2, Unknown (22):
  { [5 bytes data]
• TLSv1.2, TLS handshake, Finished (20):
  { [16 bytes data]
• SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
• ALPN, server did not agree to a protocol
• Server certificate:

•    subject: ...
  

•    start date: 2014-06-24 08:00:33 GMT
  

•    expire date: 2015-06-24 08:00:33 GMT
  

•    issuer: ...
  

•    SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  

• SSLv2, Unknown (23):
  } [5 bytes data]

  GET /repos/websvn/ HTTP/1.1
  Host:
  User-Agent: curl/7.42.0-DEV
  Accept: /

• SSLv2, Unknown (23):
  { [5 bytes data]
  < HTTP/1.1 401 Authorization Required
  < Date: Mon, 27 Apr 2015 12:34:58 GMT
  < Server: Apache/2.2.29 (FreeBSD) SVN/1.8.13 PHP/5.4.39 mod_ssl/2.2.29 OpenSSL/1.0.2a DAV/2
  < WWW-Authenticate: Negotiate
  < Content-Length: 553
  < Content-Type: text/html; charset=iso-8859-1
  <
• SSLv2, Unknown (23):
  { [5 bytes data]
• Ignoring the response-body
  { [553 bytes data]
• Connection #0 to host left intact
• Issue another request to this URL: 'https:///repos/websvn/'
• Found bundle for host : 0x28828c50
• Re-using existing connection! (#0) with host
• Connected to (<server_ip>) port 443 (#0)
• Server auth using Negotiate with user ''
• SSLv2, Unknown (23):
  } [5 bytes data]

  GET /repos/websvn/ HTTP/1.1
  Host:
  Authorization: Negotiate YIIPPQYGKwYBBQUCoIIPMTCCDy2gJzAlBgkqhkiG...
  User-Agent: curl/7.42.0-DEV
  Accept: /

• SSLv2, Unknown (23):
  { [5 bytes data]
  < HTTP/1.1 200 OK
  < Date: Mon, 27 Apr 2015 12:34:58 GMT
  < Server: Apache/2.2.29 (FreeBSD) SVN/1.8.13 PHP/5.4.39 mod_ssl/2.2.29 OpenSSL/1.0.2a DAV/2
  < WWW-Authenticate: Negotiate oRQwEqADCgEAoQsGCSqGSIb3EgECAg==
  < X-Powered-By: PHP/5.4.39
  < Content-Language: de
  < Content-Length: 6155
  < Content-Type: text/html; charset=UTF-8
  <
• SSLv2, Unknown (23):
  { [5 bytes data]
• Closing connection 0
• SSLv2, Unknown (21):
  } [5 bytes data]
• TLSv1.2, TLS alert, Client hello (1):
  } [2 bytes data]

Both JGSS (Apache Tomcat) and MIT Kerberos (Apache Web Server) still
respond with a token. So, either way one needs to pass the token back.

—
Reply to this email directly or view it on GitHub
#232 (comment).

[iboukris]

On Mon, Apr 27, 2015 at 12:57 PM, Michael Osipov [email protected]
wrote:

@Frenche https://github.com/frenche, please read comments 242 to 244 in
he Mozilla issue. They have decided to disable mutual auth. This is the
right thing to do if you cannot do it right.

I've read it.

The problem is not SPNEGO. Generally, GSS-API mechanisms are intended to
be complete before client communication starts. This works perfectly with
SASL. Unfortunately, HTTP is crappy here. It is not connection-oriented and
mixes authentication with client communication. This is our (serious)
problem, not the mechanism. More over, the response token can be on any
status, not only 200. It can be 2xx, 3xx or 4xx.

The simple answer why people don't get it right is because they don't read
the RFCs and simply do not comprehend how context establishment is support
to work. I see this too often on Stack Overflow and other sites. (I agree
that this is hard stuff)

I usually prefer to test and figure out read blogs, then RFC is fun to read
because it answers your questions..

Why it matters to me? If curl aims to be best-in-class (which it is to
me), I'd expect to have best-in-class auth support. I/we gained so much
from curl that I feel obliged to give much back.

I really much want to contribute to 'libcurl' in this area and in fact I am
working on it.
I'm just a bit slow.., it all started a few weeks ago when I noticed
'libcurl' sends two empty requests with '--anyauth' (when negotiate is
chosen) since then I found several related stuff to do in 'libcurl', some
fixes on server side 'mod_gss_api' and even fixes in 'gss-ntlmssp'
library...

Hope I'll have something ready to share soon (at least a working draft -
for which feedback and test will be much appreciated) an I wish it would
get integrated.
Either way however, I am quite happy about all that as I feel I have
learned a lot on the process.

[michael-o]

Am 2015-04-27 um 15:21 schrieb Isaac Boukris:

On Mon, Apr 27, 2015 at 12:57 PM, Michael Osipov [email protected]
wrote:

@Frenche https://github.com/frenche, please read comments 242 to 244 in
he Mozilla issue. They have decided to disable mutual auth. This is the
right thing to do if you cannot do it right.

I've read it.

The problem is not SPNEGO. Generally, GSS-API mechanisms are intended to
be complete before client communication starts. This works perfectly with
SASL. Unfortunately, HTTP is crappy here. It is not connection-oriented and
mixes authentication with client communication. This is our (serious)
problem, not the mechanism. More over, the response token can be on any
status, not only 200. It can be 2xx, 3xx or 4xx.

The simple answer why people don't get it right is because they don't read
the RFCs and simply do not comprehend how context establishment is support
to work. I see this too often on Stack Overflow and other sites. (I agree
that this is hard stuff)

I usually prefer to test and figure out read blogs, then RFC is fun to read
because it answers your questions..

Though, reading RFC is sometimes hard, I prefer that over blogs because
people describe it even worse...

Why it matters to me? If curl aims to be best-in-class (which it is to
me), I'd expect to have best-in-class auth support. I/we gained so much
from curl that I feel obliged to give much back.

I really much want to contribute to 'libcurl' in this area and in fact I am
working on it.
I'm just a bit slow.., it all started a few weeks ago when I noticed
'libcurl' sends two empty requests with '--anyauth' (when negotiate is
chosen) since then I found several related stuff to do in 'libcurl', some
fixes on server side 'mod_gss_api' and even fixes in 'gss-ntlmssp'
library...

Hope I'll have something ready to share soon (at least a working draft -
for which feedback and test will be much appreciated) an I wish it would
get integrated.
Either way however, I am quite happy about all that as I feel I have
learned a lot on the process.

There is no need to rush. Take your time and ping me if you have something to test. You maybe want to branch off master and try around. I would clone and test.

If you are looking for more server-side acceptors, you my try my fork of mod_spnego written by a Heimdal developer and my SPNEGO authenticator for Apache Tomcat.

[iboukris]

Thanks Michael.

@bagder here is a shorter version which reproduce it with basic auth against IIS

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <curl/curl.h>

int main(void)
{
  CURL *curl;
  CURLcode res;
  char *data = (char *) calloc(1, 2500);

  curl = curl_easy_init();
  if(curl) {
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
    curl_easy_setopt(curl, CURLOPT_URL, "http://ms.frenche.cp/mika/");
    curl_easy_setopt(curl, CURLOPT_USERPWD, "usera:passa");
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC | CURLAUTH_ONLY);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, 2500);

    res = curl_easy_perform(curl);
  }
  curl_global_cleanup();
  return 0;
}

Results:

* STATE: INIT => CONNECT handle 0x8e298b4; line 1048 (connection #-5000)
* Added connection 0. The cache now contains 1 members
*   Trying 10.0.0.200...
* STATE: CONNECT => WAITCONNECT handle 0x8e298b4; line 1101 (connection #0)
* Connected to ms.frenche.cp (10.0.0.200) port 80 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x8e298b4; line 1197 (connection #0)
* Marked for [keep alive]: HTTP default
* STATE: SENDPROTOCONNECT => DO handle 0x8e298b4; line 1215 (connection #0)
> POST /mika/ HTTP/1.1
Host: ms.frenche.cp
Accept: */*
Content-Length: 2500
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue

* STATE: DO => DO_DONE handle 0x8e298b4; line 1305 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x8e298b4; line 1432 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x8e298b4; line 1445 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 401 Unauthorized
< Content-Length: 1656
< Content-Type: text/html
* Server Microsoft-IIS/6.0 is not blacklisted
< Server: Microsoft-IIS/6.0
< WWW-Authenticate: Negotiate
< WWW-Authenticate: NTLM
< WWW-Authenticate: Basic realm="frenche.cp"
< X-Powered-By: ASP.NET
< Date: Mon, 27 Apr 2015 20:46:42 GMT
* Marked for [closure]: Mid-auth HTTP and much data left to send
<
* Excess found in a non pipelined read: excess = 1656 url = /mika/ (zero-length body)
* Closing connection 0
* The cache now contains 0 members
* Issue another request to this URL: 'http://ms.frenche.cp/mika/'
* STATE: PERFORM => CONNECT handle 0x8e298b4; line 1591 (connection #-5000)
* Added connection 1. The cache now contains 1 members
* Hostname ms.frenche.cp was found in DNS cache
*   Trying 10.0.0.200...
* STATE: CONNECT => WAITCONNECT handle 0x8e298b4; line 1101 (connection #1)
* Connected to ms.frenche.cp (10.0.0.200) port 80 (#1)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x8e298b4; line 1197 (connection #1)
* Marked for [keep alive]: HTTP default
* STATE: SENDPROTOCONNECT => DO handle 0x8e298b4; line 1215 (connection #1)
* Server auth using Basic with user 'usera'
> POST /mika/ HTTP/1.1
Host: ms.frenche.cp
Authorization: Basic dXNlcmE6cGFzc2E=
Accept: */*
Content-Length: 2500
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue

* STATE: DO => DO_DONE handle 0x8e298b4; line 1305 (connection #1)
* STATE: DO_DONE => WAITPERFORM handle 0x8e298b4; line 1432 (connection #1)
* STATE: WAITPERFORM => PERFORM handle 0x8e298b4; line 1445 (connection #1)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 100 Continue
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Date: Mon, 27 Apr 2015 20:46:42 GMT
* Server Microsoft-IIS/6.0 is not blacklisted
< Server: Microsoft-IIS/6.0
< X-Powered-By: ASP.NET
< X-AspNet-Version: 1.1.4322
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Content-Length: 94
<
<html>
<body>
<img src="h.jpg">
<h2>Hello!</h2>
<p> FRENCHE\usera </p>
</body>
</html>
* STATE: PERFORM => DONE handle 0x8e298b4; line 1615 (connection #1)
* Connection #1 to host ms.frenche.cp left intact
* Expire cleared

[michael-o]

I can confirm one of Isaac's statements. Just issued a few requests to an IIS instance with curl and --next and I see that the server says Persistent-Auth: false.

As we know, curl closes the connection after successful authentication. This is a pain if the communiction is TLS-protected. The TLS session has to be re-established over and over again.

[bagder]

in @Frenche's example above I cannot see any problem. Can you? I mostly read you reacting to the specific wording in that info message, but other than that?

curl closes the connection instead of wasting time downloading data it only discards.

[bagder]

@michael-o TLS or not, it acts the same way on this

[iboukris]

Hi,

On Sun, May 10, 2015 at 5:32 PM, Daniel Stenberg [email protected]
wrote:

in @Frenche https://github.com/frenche's example above I cannot see any
problem. Can you? I mostly read you reacting to the specific wording in
that info message, but other than that?

curl closes the connection instead of wasting time downloading data it
only discards.

Indeed after reviewing the specs about "Expect: 100 continue" the closure
of the connection seem less strange.

If I understand correctly the only alternative the client has when it
receives a final status code is to send all the data (to be discarded by
the server) so if it is a lot of data it may be better to close the
connection instead.

[bagder]

Nothing seems wrong here from what I can see. Closing.

[michael-o]

@bagder, let me get this straight: this warning/error is not a bug but normal behavior?

[bagder]

It is just information and is expected behavior, yes.