SIGSEV: mbedtls::Download - I believe you're accessing a non-allocated memory chunk? about minihttp

[IngwiePhoenix]

Oh by the way, error only occurs on HTTPS - never on HTTP.

[fgenesis]

I will look into this fully when i have time, but won't have time until early april unfortunately...

From a quick glance google.com does a weird redirect sorta thing (sets a cookie for google.at in my case). I get the following line of trace print: "_ParseHeader: Not chunked transfer and content-length==0, this will go fail".
Explains why it crashes. But i have no idea why google is sending this weird HTTP reply.
Will push something real quick to fix the crash at least, but a better fix has to wait until later.

[fgenesis]

What. google.com sends this in the header:

alternate-protocol: 443:quic,p=1
alt-svc: quic=":443"; ma=2592000; v="31,30,29,28,27,26,25"``

Maybe there is a special header that has to be sent that quic is not supported but i can't imagine that, since HTTP is all about backwards compatibility.
Seems to be an issue specific to google, at least it didn't happen for me with any other site i tested. Might be related to a mbedtls misconfiguration on client side that google is specifically detecting, but i doubt this.

What really confuses me is that google replies with some headers but an empty body (which is why neither content-length not chunked transfer encoding is set. Makes sense but i haven't seen this sort of reply before.)

Right now i have no idea how to make the server happy to send an actual body. Try to fake a user-agent of a popular browser? Just did a cross-check with curl and that works.

[IngwiePhoenix]

I sadly didn't find out a good way to get Google to work, but found out in the process that https://api.github.com/zen also does not work - it results in the contents being NULL, triggering the "Error!" message in example 1. Any idea for that one?

[IngwiePhoenix]

Actually nevermind... that was totally an issue on my end. I ended up getting it working. :)

I had to set a User-Agent here as well. Example code:

// Example 1: Dead-simple one-shot file download API

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>

#include "minihttp.h"

using namespace minihttp;

class DLSocket : public HttpSocket
{
public:
    DLSocket() : buf(NULL), bufsz(0), bufcap(0), finished(false), fail(false){}

    virtual ~DLSocket() {}

    char *buf;
    size_t bufsz;
    size_t bufcap;
    bool finished;
    bool fail;

protected:

    void _OnRequestDone()
    {
        finished = true;
        buf[bufsz] = 0; // zero-terminate
    }

    void _OnRecv(void *incoming, unsigned size)
    {
        if(!size || !IsSuccess())
            return;
        if(bufcap + size + 1 >= bufsz) // always make sure there's 1 more byte free for the zero-terminator
        {
            bufcap += (bufcap / 2) + size + 1;
            buf = (char*)realloc(buf, bufcap);
            if(!buf)
            {
                fail = true;
                close();
            }
        }
        memcpy(buf + bufsz, incoming, size);
        bufsz += size;
    }
};

int main(int argc, char *argv[]){
    char* url = "https://api.github.com/zen";

    if(!InitNetwork())
        return 2;

    DLSocket dl;
    dl.SetBufsizeIn(64 * 1024);
    dl.SetNonBlocking(false);
    dl.SetFollowRedirect(true);
    dl.SetAlwaysHandle(false);
    //dl.SetUserAgent("curl/7.40.0");
    dl.Download(url, NULL, NULL, NULL);

    while(dl.isOpen() || dl.HasPendingTask())
        dl.update();

    if(!dl.finished || dl.fail)
    {
        free(dl.buf);
        return 2;
    }

    puts(dl.buf);
    free(dl.buf);
    return 0;
}

Consider giving the minihttp::Download api an argument to set the user-agent?

[fgenesis]

No arg, just a default. That function shouldn't be used anyway ;)
Thanks for the investigation so far.

EDIT: I have no idea why (or if) the User-Agent field is required. Don't think the spec mandates it...