problem with CURLOPT_POSTFIELDS not working about curlcpp

[JosephP91]

You should write:

And it works. Let me know if you still have problems.

[pixiepoop]

Hello!
Still the same issue I'm afraid. For the record I'm using gcc 4.8.2

The problem is with the specialisation of cur_pair<T, string> in curl_pair.h. This takes a std::string and then when calling second() returns a raw pointer to the string data.

When the curl_pair<T, string> object goes out of scope (just after the easy.add() call returns) the string object is destroyed and the raw pointer is now invalid, but I think this data is actually read in the curl_perfom() call.

So when calling easy.add(curl_pair<CURLoption, string>(CURLOPT_POSTFIELDS, "lol"));
the following happens:

1. std::string object is created for "lol"
2. the curl_pair<CURLOption, string> copies this to its std::string "value" member
3. easy.add() calls curl_pair.second() which takes a raw pointer to the "value" member, which is passed to curl_easy_setopt().
4. when curl.add() returns the curl_pair object goes out of scope and the std::string "value" member is destroyed.

Later when curl.perform() is called this memory is read, and you might (or might not) be lucky and get the original (correct) data which the string stored, which is why it might apparently work sometimes.

Incidentally I think that there is probably a similar issue with the specialisation of cur_pair for curl_form, curl_header etc.

The only way I can see to fix this (with the same interface) would be to somehow hold onto the string (and other) objects until curl_perform() is called. e.g. have a vector of strings which could be freed after curl_perform() has been called.

I can try to give you the simplest complete example which fails if you like.

[JosephP91]

Yeah, I'd like to have your example so I can try it :) Thank you!

[pixiepoop]

#include "curl_easy.h"
int main()
{
      curl::curl_easy easy;

      std::string postPayload("username=user123&password=pass123");

      easy.add(curl_pair<CURLoption,string>(CURLOPT_URL,"http://posttestserver.com/post.php") );
      easy.add(curl_pair<CURLoption, string>(CURLOPT_POSTFIELDS,  "username=user123&password=pass123"));
      //easy.add(curl_pair<CURLoption, string>(CURLOPT_POSTFIELDS,  postPayload)); // This line will work correctly.....
      easy.add(curl_pair<CURLoption, bool>(CURLOPT_POST,  true));
      easy.perform();

      return 0;
}

Compile options with gcc 4.8.2...
g++ -std=c++11 -g -Wall main.cpp -Ibuild/src -Lbuild/src -lcurlcpp -lcurl

As it is I get:
"Successfully dumped 0 post variables."

With the comment out line instead I get
"Successfully dumped 2 post variables."

You can also check the output at the site "http://posttestserver.com".

Thanks for all the hard work!

[JosephP91]

I figured it out. I think the problem is the pass-by-value in curl_pair.h (as you described in the last comment). Now the values are passed by reference and your example works dumping succesfully 2 variables. Can you try? Let me know! I updated the repository.

[pixiepoop]

Still no luck I'm afraid - it still fails with gcc 4.8 on linux.

This will still have the same fundamental problem. It is likely that it might appear to work under some conditions.

Try using valgrind on the example above, and you will see the program attempting to access freed memory. (see attached image)

Please read http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
Strings passed to libcurl as 'char *' arguments, are copied by the library; thus the string storage associated to the pointer argument may be overwritten after curl_easy_setopt() returns. The only exception to this rule is really CURLOPT_POSTFIELDS

This is what is happening:
This line:

easy.add(curl_pair<CURLoption, string>(CURLOPT_POSTFIELDS,  "username=user123&password=pass123"));

1. A std::string object is created, allocating some memory somewhere (let's say 0x12345678) and writes "username=user123&password=pass123" to this memory location.
2. The curl_pair<CURLoption, string> is creating storing a reference to this string object (this is fine at this stage).
3. easy.add() is called this calls:

const CURLcode code = curl_easy_setopt(this->curl,pair.first(),pair.second());

pair.second() gets the pointer to the raw string data (0x12345678) and passes this into curl_easy_setopt. (again this is fine at this stage). _BUT THE STRING DATA IS NOT COPIED AT THIS STAGE_ (at least according to http://curl.haxx.se/libcurl/c/curl_easy_setopt.html)

1. the call to easy.add() returns
2. The original std::string object is now destroyed and the memory at location 0x12345678 is freed (even though you might get 'lucky' and the data is not overwritten)
3. When easy.perfrom() is eventually called, it attempts to access the string at location 0x12345678 which is invalid (although may still contain the correct data).

Unfortunately it looks like we have to store the string internally somehow until easy.perform() is called, or maybe force the CURLOPT_COPYPOSTFIELDS to be used instead.

[JosephP91]

You're right. Reading again the documentation about CURLOPT_COPYPOSTFIELDS and using it, the program seems to dump both the variables. Do you think we should force COPYPOSTFIELDS usage or find another way to fix this?

[pixiepoop]

Changing the specialisation of curl_pair in curl_pair.h as below fixes the issue, and only copies when required.

Reading the documentation, the only trouble with COPYPOSTFIELDS will be if the string is not null terminated - this shouldn't be a problem since we are getting the string from std::string which should ensure that the raw buffer is null terminated.

template<class T> class curl_pair<T,string> {
    public:
        /**
         * The two parameters constructor gives users a fast way to
         * build an object of this type.
         */
        curl_pair(const T option, const string &value) : option(option), value(value) {};
        curl_pair(const T option, string &&value) : option(option == CURLOPT_POSTFIELDS ? CURLOPT_COPYPOSTFIELDS : option ), value(value) {};
        /**
         * Simple method that returns the first field of the pair.
         */
        inline const T first() const noexcept {
            return this->option;
        }
        /**
         * Simple method that returns the second field of the pair as
         * a C string, so a const char *.
         */
        inline const char *second() const noexcept {
            return this->value.c_str();
        }
    private:
        const T option;
        const string &value;
    };

Only works for C++11 (which I'm guessing you are using anyway given the noexcept...)
This gives the best of both worlds - it will only copy the buffer if you supply an 'rvalue', otherwise the string is not copied.

You could still break it with something like:

#include "../curlcpp/include/curl_easy.h"
int main()
{
      curl::curl_easy easy;
      {
        std::string postPayload("username=user123&password=pass123");

        easy.add(curl_pair<CURLoption, string>(CURLOPT_URL, "http://posttestserver.com/post.php"));
        easy.add(curl_pair<CURLoption, string>(CURLOPT_POSTFIELDS,  postPayload));
        easy.add(curl_pair<CURLoption, bool>(CURLOPT_POST,  true));
      }
      easy.perform();

      return 0;
}

If you want to be able to handle this case too, then you will have to always copy the string (e.g. just replace CURLOPT_POSTFIELDS with CURLOPT_COPYPOSTFIELDS

[JosephP91]

Yes, I think I will go for the first solution. Thank you @pixiepoop !