Unordered loading (XML and JSON) not working when types have a wrapper (e.g. ptr_wrapper) about cereal

[Devacor]

I should mention I am on the head of "develop"

[Devacor]

Minimal test included!

#include <iostream>
#include <sstream>
#include <string>
#include <map>

#include "cereal/cereal.hpp"
#include "cereal/types/map.hpp"
#include "cereal/types/vector.hpp"
#include "cereal/types/memory.hpp"
#include "cereal/types/string.hpp"
#include "cereal/types/base_class.hpp"

#include "cereal/archives/json.hpp"
#include <cereal/types/polymorphic.hpp>

class BaseClass : public std::enable_shared_from_this<BaseClass> {
public:
    virtual ~BaseClass(){}

    template <class Archive>
    void serialize(Archive & archive){
        archive(CEREAL_NVP(name), CEREAL_NVP(baseMember));
    }
protected:
    BaseClass(const std::string &a_name):
        name(a_name){
    }

    std::string name;
    int baseMember; //let this have random junk so we can see if it saves right.
};

class DerivedClass : public BaseClass {
    friend cereal::access;
public:
    static std::shared_ptr<DerivedClass> make(const std::string &a_name, int a_derivedMember){
        return std::shared_ptr<DerivedClass>(new DerivedClass(a_name, a_derivedMember));
    }

    template <class Archive>
    void serialize(Archive & archive){
        archive(CEREAL_NVP(derivedMember), cereal::make_nvp("base", cereal::base_class<BaseClass>(this)));
    }
private:
    DerivedClass(const std::string &a_name, int a_derivedMember):
        BaseClass(a_name),
        derivedMember(a_derivedMember){
    }

    template <class Archive>
    static DerivedClass * load_and_allocate(Archive &archive){
        int derivedMember;
        archive(CEREAL_NVP(derivedMember));
        DerivedClass* object = new DerivedClass("", derivedMember);
        archive(cereal::make_nvp("base", cereal::base_class<BaseClass>(object)));
        return object;
    }

    int derivedMember;
};

CEREAL_REGISTER_TYPE(DerivedClass);

void saveTest(){
    std::stringstream stream;
    {
        cereal::JSONOutputArchive archive(stream);
        auto testSave = DerivedClass::make("TestName", 4);
        archive(cereal::make_nvp("test", testSave));
    }
    std::cout << stream.str() << std::endl;
    std::shared_ptr<DerivedClass> loaded;
    {
        cereal::JSONInputArchive archive(stream);
        archive(cereal::make_nvp("test", loaded));
    }
    std::stringstream stream2;
    {
        cereal::JSONOutputArchive archive(stream2);
        archive(cereal::make_nvp("test", loaded));
    }
    std::cout << stream2.str() << std::endl;
    std::cout << "TA-DA!" << std::endl;
}

int main(){
    saveTest();
}

[Devacor]

Stack Overflow question for points if you like: http://stackoverflow.com/questions/20940919/c11-cereal-load-and-allocate-not-loading-correctly

[Devacor]

New minimal test! It seems like even the most basic example fails. I was looking for coverage in unit tests and I didn't see any instance of load_and_allocate being used.* (edit: it seems this stuff is in the sandbox cpp files, but in these examples there are no member values being loaded and so this case would not be hit.) I tried this with XML archives as well with the same problem (though instead the exception throws from xml.hpp startNode()).

#include <iostream>
#include <sstream>
#include <string>
#include <map>

#include "cereal/cereal.hpp"
#include "cereal/types/map.hpp"
#include "cereal/types/vector.hpp"
#include "cereal/types/memory.hpp"
#include "cereal/types/string.hpp"
#include "cereal/types/base_class.hpp"

#include "cereal/archives/json.hpp"

class BasicClass {
    friend cereal::access;
public:
    static std::shared_ptr<BasicClass> make(const std::string &a_name, int a_member){
        return std::shared_ptr<BasicClass>(new BasicClass(a_name, a_member));
    }

    template <class Archive>
    void serialize(Archive & archive){
        archive(CEREAL_NVP(name), CEREAL_NVP(member));
    }
private:
    BasicClass(const std::string &a_name, int a_member):
        name(a_name),
        member(a_member){
    }

    template <class Archive>
    static BasicClass * load_and_allocate(Archive &archive){
        std::string name;
        int member;
        archive(CEREAL_NVP(name), CEREAL_NVP(member));
        BasicClass* object = new BasicClass(name, member);
        return object;
    }

    std::string name;
    int member;
};

void saveTest(){
    std::stringstream stream;
    {
        cereal::JSONOutputArchive archive(stream);
        auto testSave = BasicClass::make("TestName", 4);
        archive(cereal::make_nvp("test", testSave));
    }
    std::cout << stream.str() << std::endl;
    std::shared_ptr<BasicClass> loaded;
    {
        cereal::JSONInputArchive archive(stream);
        archive(cereal::make_nvp("test", loaded));
    }
    std::stringstream stream2;
    {
        cereal::JSONOutputArchive archive(stream2);
        archive(cereal::make_nvp("test", loaded));
    }
    std::cout << stream2.str() << std::endl;
    std::cout << "TA-DA!" << std::endl;
}

int main(){
    saveTest();
}

[AzothAmmo]

Definitely a bug on our part introduced with the out of order loading. Probably also exists for XML archives. The bug is exactly in the search function where you described, and the issue is that we have a few special wrappers we sometimes use that introduce additional nodes into the JSON/XML trees that are cereal-specific metadata and not really relevant to a user looking at the JSON/XML.

Toying around two ideas to fix this in my head right now. One is to adjust the search function or introduce some state to the archive such that it knows when it is inside of a wrapper. The other solution would be to split cereal metadata into an entirely different node than actual data saved in a JSON/XML archive.

So for your example (slightly changed to return a shared_ptr<BaseClass> instead of DerivedClass to force use of the polymorphic stuff, and also modified to output exactly the same pointer twice in a row (with names nvp1 and nvp2):

{
  "cereal_data": {
    "nvp1": {
      "derivedMember": 4,
      "base": {
        "name": "TestName",
        "baseMember": 0
      }
    }
  },
  "cereal_metadata_do_not_edit":
  {
    "nvp1": {
      "polymorphic_id": 2147483649,
      "polymorphic_name": "DerivedClass",
      "ptr_id": 2147483649
    }
    "nvp2": {
      "polymorphic_id": 1,
      "ptr_id": 1
    }
  }
}

I'll have to think this over. The second approach would appeal to some others (e.g. DrAWolf) who have been asking for the text based archives to be more readable, and this would solve that issue more or less. We would likely have some decreased performance though.

[Devacor]

Yeah, my initial reaction was that we would have to dig the "data" part out of the archive and pass that through in our load_andor_allocate calls in line 143 and 166 in memory.hpp:

      ptr.reset( detail::Load<T, Archive>::load_andor_allocate( ar ) );

But I'm not really familiar with how you would do that.

I can see the merit to breaking out the structure as well, but I suspect that's a riskier and bigger task.

[AzothAmmo]

I'm not sure there is actually any dependence on load_and_allocate, I think the bug is just to do with the wrappers.

[Devacor]

Yeah, I'm kind of thrashing for an explanation because I am just not very familiar with this code base and in fact am kind of new to enable_if and template metaprogramming stuff that you guys employ a lot of. It's a very nested code base so I could be way off.

I guess I just see the memory.hpp save methods for PtrWrapper explicitly save data in a "data" field, but never explicitly pull the "data" field out in the corresponding load methods.

[Devacor]

For now, to work around the issue I added a line 144 in memory.hpp (as it appears on line 168 in the case of no load_and_allocate which means that there is a default constructor.)

  ar( *ptr );

I will simply avoid using the load_and_allocate archive directly and will use my serialization functions. In my load_and_allocate method I will construct an object with "default" like information.

[DrAWolf]

Hey, just as I saw the relation to my suggestions, you already mention my name.
BUT it's not what you infer.
I STRONGLY disagree with the split data approach you are pondering.
Duplication/duplicity is the root of all evil - as we all (should) know.

[DrAWolf]

The optimal solution would be a loader that could identify/recognize the polymorphic type - by looking at the entries present/given in a JSON object.

[AzothAmmo]

Actually looking at this the issue is with load_and_allocate, which does not enter into the data node created by the ptr_wrapper. In regards to the possible splitting of the data in the text archives, this doesn't duplicate any data except for NVP names to index the information. One node would contain data the other metadata. There is no way for us to avoid including this metadata somewhere. I think the only two good solutions are to keep it as is (which I don't think is such a big deal), or the alternative path.

[AzothAmmo]

Marking as closed let me know if issue persists.