Error With Nested Circular shared_ptr and weak_ptr References about cereal

[AzothAmmo]

Part of this seems to be a bug with our std::weak_ptr serialization. If you make your weak_ptr<ChildWithPointerToParent> into an std::shared_ptr<>, you can compile properly without needing the workarounds you are using. Not sure about the circular part yet, I'll tackle this initial bug first. I'm glad you are using pointers in a sufficiently complicated way to find these errors.

[AzothAmmo]

Really small test case with one portion of the issue:

#include <cereal/types/memory.hpp>
#include <cereal/archives/json.hpp>
#include <sstream>

struct C
{
  template <class Archive>
  void serialize( Archive & ar )
  { }
};

int main()
{
  std::stringstream ss;
  {
    cereal::JSONOutputArchive ar( ss );

    std::weak_ptr<C> ptr;
    ar( ptr );
  }

  return 0;
}

Fails to compile due to static asserts about serialization functions. Making the weak into a shared or a unique eliminates the problem.

[AzothAmmo]

So the other issue (based on your test case) occurs because of the ownership semantics on weak_ptrs. When we serialize one of those, we lock it and get a shared_ptr. When we do the loading, we load it up as a shared_ptr and then store it as a shared_ptr<void> in a map inside of the archive. So at this point the typed shared_ptr only exists temporarily within the weak_ptr loading code. When we also store a copy within our map, the reference count for this shared_ptr is 2.

When the temporary shared_ptr used for loading goes out of scope (and we return a weak_ptr that you requested), the reference count is now at 1 (the copy that still exists within our map of shared_ptr<void> used to prevent duplicate loads). When the archive itself goes out of scope, this last solid reference to the data is removed, so the weak_ptr you are using is now expired. This is why when you try to serialize the data you have loaded, you see that the contents for the weak_ptr just contain an id of 0, indicating nullptr.

If this is the issue you are talking about, it is something you have to fix by making sure you actually keep a stable reference to things you have weak_ptrs to. We can't do this for you since a weak_ptr fundamentally does not own its content.

[Devacor]

I think there is an error. Can you read the following? From my viewpoint I would expect different results than what cereal is giving me. This may be a hard bug to fix, but I do believe it is a bug.

Full source included at the end.

User Story: Run this code:

    std::stringstream stream;
    {
        cereal::JSONOutputArchive archive(stream);
        std::shared_ptr<BaseClass> base = DerivedClass::make("TestName", 4);
        std::shared_ptr<ChildWithPointerToParent> child = ChildWithPointerToParent::make(base);
        archive(cereal::make_nvp("test", base));
    }
    std::cout << stream.str() << std::endl;

The expected behaviour should result in correct linking to all references. They are in memory, abstraction of how you handle node traversal might make implementation difficult, but since it is being represented in memory and all instances are viable at the time of saving it should be serializable.

This means that even though we are saving a std::shared_ptr (of actual type std::shared_ptr, a needless detail) and upon loading that std::shared_ptr its std::weak_ptr reference will die in a fire and be .expired(), that isn't actually a crash bug until we actually try to dereference the weak_ptr. It's probably not what was intended by the user (to load a weak_ptr with no shared_ptr handles), but you can see why this is an actual bug in cereal.

ACTUAL:

{
    "test": {
        "polymorphic_id": 100,
        "polymorphic_name": "DerivedClass",
        "ptr_wrapper": {
            "id": 200,
            "data": {
                "derivedMember": 4,
                "base": {
                    "name": "TestName",
                    "baseMember": 0,
                    "child": {
                        "locked_ptr": {
                            "id": 200,
                            "ptr_wrapper": {
                                "id": 300,
                                "data": {
                                    "parent": {
                                        "polymorphic_id": 1, //wat? why?
                                        "ptr_wrapper": {
                                            "id": 1 //wat? why?
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

DESIRED:

{
    "test": {
        "polymorphic_id": 100,
        "polymorphic_name": "DerivedClass",
        "ptr_wrapper": {
            "id": 200,
            "data": {
                "derivedMember": 4,
                "base": {
                    "name": "TestName",
                    "baseMember": 0,
                    "child": {
                        "locked_ptr": {
                            "id": 200,
                            "ptr_wrapper": {
                                "id": 300,
                                "data": {
                                    "parent": {
                                        "polymorphic_id": 100,
                                        "ptr_wrapper": {
                                            "id": 100
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

So obviously loading breaks. The save doesn't even work, so no need to point that out, but we would expect the load to create the DerivedClass with its shitty weak_ptr, and the archive would keep a shared_ptr to that constructed child until the archive exits at which point nothing has a handle on the weak_ptr and it expires. There should at no point be a crash in the load process, only upon trying to use that weak_ptr handle that was loaded, but immediately died after exiting cereal.

---

Now for the juicy bit. So yes, my example was dumb because I was saving a shared_ptr that had a weak_ptr which would obviously expire.

My actual setup is not really like that. It is the opposite. I save the shared_ptr (in this case, the ChildWithPointerToParent is the TextureHandle, and the BaseClass is my TextureDefinition. A rectangle or drawable object would own the TextureHandle, and I want to save that rectangle, so it should save its handle which should have a strong shared_ptr ref to the TextureDefinition. This should all nicely load but does not.)

So what happens if we change the code:

    std::stringstream stream;
    {
        cereal::JSONOutputArchive archive(stream);
        std::shared_ptr<BaseClass> base = DerivedClass::make("TestName", 4);
        std::shared_ptr<ChildWithPointerToParent> child = ChildWithPointerToParent::make(base);
        archive(cereal::make_nvp("test", child));
    }
    std::cout << stream.str() << std::endl;

Identical except now we save the child. Now, consider this. The child has a shared_ptr ref to the BaseClass, so by saving the child we would expect there to be the local scope of the archive load call to dump contents into a shared_ptr (refcount of 1 for the child), and the child should have a shared_ptr to the base (refcount of 1 for the base), and so everything SHOULD be okay.

But it is not. Why? Because it doesn't save correctly.

ACTUAL:

{
    "test": {
        "id": 100,
        "ptr_wrapper": {
            "id": 200,
            "data": {
                "parent": {
                    "polymorphic_id": 200,
                    "polymorphic_name": "DerivedClass",
                    "ptr_wrapper": {
                        "id": 300,
                        "data": {
                            "derivedMember": 4,
                            "base": {
                                "name": "TestName",
                                "baseMember": 0,
                                "child": {
                                    "locked_ptr": {
                                        "id": 100,
                                        "ptr_wrapper": {
                                            "id": 1  //wat? why?
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

DESIRED:

{
    "test": {
        "id": 100,
        "ptr_wrapper": {
            "id": 200,
            "data": {
                "parent": {
                    "polymorphic_id": 200,
                    "polymorphic_name": "DerivedClass",
                    "ptr_wrapper": {
                        "id": 300,
                        "data": {
                            "derivedMember": 4,
                            "base": {
                                "name": "TestName",
                                "baseMember": 0,
                                "child": {
                                    "locked_ptr": {
                                        "id": 100,
                                        "ptr_wrapper": {
                                            "id": 100
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

#include <iostream>
#include <sstream>
#include <string>
#include <map>

#include "cereal/cereal.hpp"
#include "cereal/types/map.hpp"
#include "cereal/types/vector.hpp"
#include "cereal/types/memory.hpp"
#include "cereal/types/string.hpp"
#include "cereal/types/base_class.hpp"

#include "cereal/archives/json.hpp"
#include <cereal/types/polymorphic.hpp>

class ChildWithPointerToParent;

class BaseClass : public std::enable_shared_from_this<BaseClass> {
public:
    virtual ~BaseClass(){}

    template <class Archive>
    void serialize(Archive & archive){
        archive(CEREAL_NVP(name), CEREAL_NVP(baseMember), CEREAL_NVP(child)); //crashes here!
    }

    void addChild(std::shared_ptr<ChildWithPointerToParent> a_child){
        child = a_child;
    }
protected:
    BaseClass(const std::string &a_name):
        name(a_name){
    }

    std::weak_ptr<ChildWithPointerToParent> child;
    std::string name;
    int baseMember; //let this have random junk so we can see if it saves right.
};

class DerivedClass : public BaseClass {
    friend cereal::access;
public:
    static std::shared_ptr<DerivedClass> make(const std::string &a_name, int a_derivedMember){
        return std::shared_ptr<DerivedClass>(new DerivedClass(a_name, a_derivedMember));
    }

    template <class Archive>
    void serialize(Archive & archive){
        archive(CEREAL_NVP(derivedMember), cereal::make_nvp("base", cereal::base_class<BaseClass>(this)));
    }
private:
    DerivedClass(const std::string &a_name, int a_derivedMember):
        BaseClass(a_name),
        derivedMember(a_derivedMember){
    }

    template <class Archive>
    static DerivedClass * load_and_allocate(Archive &archive){
        return new DerivedClass("", 0); //values loaded in serialize (using work-around in memory.hpp)
    }

    int derivedMember;
};

class ChildWithPointerToParent {
    friend cereal::access;
public:
    virtual ~ChildWithPointerToParent(){} //Why do I have to do this?  I get an error if I don't.
    static std::shared_ptr<ChildWithPointerToParent> make(std::shared_ptr<BaseClass> parent){
        auto child = std::shared_ptr<ChildWithPointerToParent>(new ChildWithPointerToParent(parent));
        parent->addChild(child);
        return child;
    }

    template <class Archive>
    void serialize(Archive & archive){
        archive(CEREAL_NVP(parent));
    }

private:
    ChildWithPointerToParent(std::shared_ptr<BaseClass> a_parent):
        parent(a_parent){
    }

    template <class Archive>
    static ChildWithPointerToParent * load_and_allocate(Archive &archive){
        return new ChildWithPointerToParent(nullptr); //values loaded in serialize (using work-around in memory.hpp)
    }
    std::shared_ptr<BaseClass> parent;
};

CEREAL_REGISTER_TYPE(DerivedClass);
CEREAL_REGISTER_TYPE(ChildWithPointerToParent); //Why do I have to do this?  I get an error if I don't.

void saveTest(){
    std::stringstream stream;
    {
        cereal::JSONOutputArchive archive(stream);
        std::shared_ptr<BaseClass> base = DerivedClass::make("TestName", 4);
        std::shared_ptr<ChildWithPointerToParent> child = ChildWithPointerToParent::make(base);
        archive(cereal::make_nvp("test", child));
    }
    std::cout << stream.str() << std::endl;
    std::shared_ptr<ChildWithPointerToParent> loadedChild;
    {
        cereal::JSONInputArchive archive(stream);
        archive(cereal::make_nvp("test", loadedChild));
    }
    std::stringstream stream2;
    {
        cereal::JSONOutputArchive archive(stream2);
        archive(cereal::make_nvp("test", loadedChild));
    }
    std::cout << stream2.str() << std::endl;
    std::cout << "TA-DA!" << std::endl;
}

int main(){
    saveTest();
}

[Devacor]

Sorry for taking a while to reply. I seriously thought hard about what I was seeing and if that was a bug on my end or not. But in the end, I don't think cereal should be saving things with an id of 1 in this case.

[Devacor]

I don't know of any work-around to save the structures I'm talking about either, except by some kind of hacky break in the circular references and some user logic to make sure parents and children add each-other properly after the load steps are done. Not ideal.

[Devacor]

One last message tonight. If we can hammer on some fixes for this, I am absolutely going to continue to utilize cereal for all of my load/save needs if possible. A bit of hammering on from some existing production code should help pick out some of the bugs, and hopefully get it closer to viable for a boost submission sometime in the future.

You guys have something special here and I may not have enough time to deeply invest in active development, but I will promote it if I can when I open source my own little engine (not that it's amazing or anything, and it isn't money in the bank, but it's still kind of cool.)

Thanks so much for responding to these in a pretty timely manner. You're the best.

[AzothAmmo]

The pointer id tags are constructed as follows: the MSB is set to 1 if this is the first time we are saving a type and 0 otherwise (we also set MSB2 in some cases dealing with polymorphism). This is why you get an id of something like 2147483649 (0x80000001) and then an id of 1 (0x1) when you serialize it a second time. We use the value of the MSB when we perform a load to determine whether we need to do a full data load or to look up the previously loaded pointer in our map, so from what I can see there isn't anything wrong with the actual output of the first round of serialization. However you should be able to load and then serialize again and get the same thing, so there is still something fishy going on. I'll keep looking.

[Devacor]

Yeah, I can see how that's the case with saving/loading the child. Saving/loading the parent however I'm not sure how the most nested pointer would look up the right object. So with that in mind, the output is something like this:

...
                                    "parent": {
                                        "polymorphic_id": 1, //wat? why?
                                        "ptr_wrapper": {
                                            "id": 1 //wat? why?
                                        }
                                    }
...

Shouldn't the polymorphic_id be some non-one value? Maybe I misunderstand still.

In the case of the save/load on the child I understand then that the save output is right. The loading sequence still breaks with the exception:

          throw Exception("Error while trying to deserialize a smart pointer. Could not find id " + std::to_string(id));

Where "id" is 1. This is why I assumed the error hapened when it hit the innermost ptr_wrapper.

Thanks for looking!

[Devacor]

I'm going to update my project to the latest as well to grab your fixes to load_and_allocate (currently working around with that silly hack)

[AzothAmmo]

This is the exact reason why your code is crashing cereal right now:

When we start loading the outer shared_ptr, it is the first time we've seen it (MSB is 1), so after doing its load, we save it in an internal map.

The problem occurs because while we are loading that value, we encounter a circular reference to ourselves and since the MSB is not 1, we expect that the value should have already been loaded and be available in the map. This fails because we can't put something in the map until the entire load is done, so an exception gets thrown when trying to load the child shared_ptr.

There's also another small bug in the load code for a shared_ptr under load_and_allocate where it is missing the registration of the id in the map, but this is trivial compared to the real issue.

[AzothAmmo]

Minimal test case:

#include <sstream>
#include <cereal/archives/json.hpp>
#include <cereal/types/memory.hpp>

struct A
{
  template <class Archive>
  void serialize( Archive & ar )
  {
    ar( ptr );
  }

  std::shared_ptr<A> ptr;
};

int main()
{
  std::stringstream ss;
  {
    cereal::JSONOutputArchive ar( ss );

    std::shared_ptr<A> a = std::make_shared<A>();
    a->ptr = a;

    ar( a );
  }

  std::cout << ss.str() << std::endl;

  std::shared_ptr<A> b;
  {
    cereal::JSONInputArchive ar( ss );
    ar( b );
  }
}

[Devacor]

Yes, nesting issues like this can be hard to deal with. Callbacks that delete the objects that call them can be similarly tricky. In those cases at least the fix can usually be done in a post step. In this case I think it's a bit harier.

Sent from my iPhone

On 2014-01-09, at 2:24 PM, Shane Grant [email protected] wrote:

Minimal test case:

#include
#include <cereal/archives/json.hpp>
#include <cereal/types/memory.hpp>

struct A
{
template
void serialize( Archive & ar )
{
ar( ptr );
}

std::shared_ptr ptr;
};

int main()
{
std::stringstream ss;
{
cereal::JSONOutputArchive ar( ss );

std::shared_ptr<A> a = std::make_shared<A>();
a->ptr = a;

ar( a );

}

std::cout << ss.str() << std::endl;

std::shared_ptr b;
{
cereal::JSONInputArchive ar( ss );
ar( b );
}
}
—
Reply to this email directly or view it on GitHub.

[AzothAmmo]

Just want to give an update on this - I haven't had any time to think about a solution yet, but will try and get something out early this week.

[Devacor]

Thanks so much. I must admit, I've been keeping my eye on this page like a hawk as I'm very interested in that update!

[AzothAmmo]

Should be fixed. There is a very minimal amount of extra overhead associated with shared_ptr now.

The solution was to defer all nested circular loads until the parent was complete, and then perform them all. This is done with a dirty flag on the map entry for that pointer id and by storing all of the deferred loads until it is safe to perform them.

[Devacor]

Thank you! I'll be taking a look at this in the next day or two!

[Devacor]

I'm having issues with this actually. if( ar.isSharedPointerValid ) in memory.hpp is throwing this at my face where it used to compile. This is with Visual Studio 2013 + the november compiler preview (REPRODUCIBLE WITH MY FIRST EXAMPLE CODE, NOT WITH YOUR STRIPPED DOWN VERSION HOWEVER):

1>  textures.cpp
1>C:\git\external\cereal\include\cereal/types/memory.hpp(171): error C3867: 'cereal::InputArchive<cereal::JSONInputArchive,0>::isSharedPointerValid': function call missing argument list; use '&cereal::InputArchive<cereal::JSONInputArchive,0>::isSharedPointerValid' to create a pointer to member
1>          C:\git\external\cereal\include\cereal/cereal.hpp(767) : see reference to function template instantiation 'void cereal::load<AA,MV::FileTextureDefinition>(Archive &,cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &> &)' being compiled
1>          with
1>          [
1>              AA=cereal::JSONInputArchive
1>  ,            Archive=cereal::JSONInputArchive
1>          ]
1>          C:\git\external\cereal\include\cereal/cereal.hpp(692) : see reference to function template instantiation 'cereal::JSONInputArchive &cereal::InputArchive<cereal::JSONInputArchive,0>::processImpl<T>(T &)' being compiled
1>          with
1>          [
1>              T=cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>
1>          ]
1>          C:\git\external\cereal\include\cereal/cereal.hpp(692) : see reference to function template instantiation 'cereal::JSONInputArchive &cereal::InputArchive<cereal::JSONInputArchive,0>::processImpl<T>(T &)' being compiled
1>          with
1>          [
1>              T=cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>
1>          ]
1>          C:\git\external\cereal\include\cereal/cereal.hpp(558) : see reference to function template instantiation 'void cereal::InputArchive<cereal::JSONInputArchive,0>::process<_Ty>(T &&)' being compiled
1>          with
1>          [
1>              _Ty=cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>
1>  ,            T=cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>
1>          ]
1>          C:\git\external\cereal\include\cereal/cereal.hpp(558) : see reference to function template instantiation 'void cereal::InputArchive<cereal::JSONInputArchive,0>::process<_Ty>(T &&)' being compiled
1>          with
1>          [
1>              _Ty=cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>
1>  ,            T=cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>
1>          ]
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(160) : see reference to function template instantiation 'ArchiveType &cereal::InputArchive<ArchiveType,0>::operator ()<cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>>(cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &> &&)' being compiled
1>          with
1>          [
1>              ArchiveType=cereal::JSONInputArchive
1>          ]
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(160) : see reference to function template instantiation 'ArchiveType &cereal::InputArchive<ArchiveType,0>::operator ()<cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &>>(cereal::memory_detail::PtrWrapper<std::shared_ptr<MV::FileTextureDefinition> &> &&)' being compiled
1>          with
1>          [
1>              ArchiveType=cereal::JSONInputArchive
1>          ]
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(150) : while compiling class template member function 'cereal::detail::InputBindingCreator<Archive,T>::InputBindingCreator(void)'
1>          with
1>          [
1>              Archive=cereal::JSONInputArchive
1>  ,            T=MV::FileTextureDefinition
1>          ]
1>          C:\git\external\cereal\include\cereal/details/static_object.hpp(56) : see reference to function template instantiation 'cereal::detail::InputBindingCreator<Archive,T>::InputBindingCreator(void)' being compiled
1>          with
1>          [
1>              Archive=cereal::JSONInputArchive
1>  ,            T=MV::FileTextureDefinition
1>          ]
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(293) : see reference to class template instantiation 'cereal::detail::InputBindingCreator<Archive,T>' being compiled
1>          with
1>          [
1>              Archive=cereal::JSONInputArchive
1>  ,            T=MV::FileTextureDefinition
1>          ]
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(290) : while compiling class template member function 'void cereal::detail::polymorphic_serialization_support<cereal::JSONInputArchive,T>::instantiate(void)'
1>          with
1>          [
1>              T=MV::FileTextureDefinition
1>          ]
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(307) : see reference to class template instantiation 'cereal::detail::polymorphic_serialization_support<cereal::JSONInputArchive,T>' being compiled
1>          with
1>          [
1>              T=MV::FileTextureDefinition
1>          ]
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(306) : while compiling class template member function 'void cereal::detail::bind_to_archives<MV::FileTextureDefinition>::bind(std::false_type) const'
1>          C:\git\external\cereal\include\cereal/details/polymorphic_impl.hpp(321) : see reference to function template instantiation 'void cereal::detail::bind_to_archives<MV::FileTextureDefinition>::bind(std::false_type) const' being compiled
1>          Source\Render\textures.cpp(13) : see reference to class template instantiation 'cereal::detail::bind_to_archives<MV::FileTextureDefinition>' being compiled
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

Could you take a look with MS Visual Studio 2013?

[Devacor]

Created a new issue: #44