Error in rtree when loading OpenStreetMap data about navitia

[pbougue]

Hi @frodrigo,
I just tried to reproduce it on Ubuntu 15.04, gcc 4.9.2, boost 1.54.0, with the paris file from metro.teczno.com, compiled in release.
I'm on commit fcca83 from today.
I was not able to reproduce :-/
Tried also with clang 3.6.0 and it worked fine.

I don't have any clue... Maybe I can try with your file?

[frodrigo]

My test include paris file from metro.teczno.com
I'm running boost 1.55.0.
Compiled in release too.

What I can do to help you to debug this ?

[pbougue]

Ok, this is strange, I will try to run it on the same platform but I believe someone in the team (out of the office now) is using it without this kind of problem.
Can you try to wipe out your compile directory and launch a new compilation to send us your compile logs ?
More random questions : Did you properly update your submodules? Even less probable with this message, did you upgrade your db (using alembic) ?

[pbougue]

Just thought of one more possible issue : did you monitor your memory during your run?

[frodrigo]

Yes. Not a problem of memory

[pbougue]

Hello, I tried running on a docker with the same configuration, debian stable, gcc 4.9.2, boost 1.55.0, same paris file in release, and I could not reproduce.
I am running out of ideas on this...
Maybe you can try to compile it with clang to see how it works (although we had runtime problems on 1.55.0 boost version from ubuntu in that case).

[pbougue]

Maybe also try in debug to see if stack trace is exploitable.

[frodrigo]

It's no compile with clang

In file included from /home/fred/navitia/source/type/type.cpp:40:
/home/fred/navitia/source/third_party/eos_portable_archive/portable_oarchive.hpp:324:15: warning: shift count >= width of type [-Wshift-count-overflow]
                                do { temp >>= CHAR_BIT; ++size; }
                                          ^   ~~~~~~~~
/usr/include/boost/archive/impl/basic_binary_oprimitive.ipp:49:19: note: in instantiation of function template specialization 'eos::portable_oarchive::save<unsigned char>' requested here
    this->This()->save(static_cast<unsigned char>(sizeof(int)));
                  ^
In file included from /home/fred/navitia/source/type/type.cpp:39:
In file included from /home/fred/navitia/source/third_party/eos_portable_archive/portable_iarchive.hpp:90:
In file included from /usr/include/boost/archive/basic_binary_iarchive.hpp:31:
/usr/include/boost/archive/detail/common_iarchive.hpp:66:9: fatal error: recursive template instantiation exceeded maximum depth of 256
        archive::load(* this->This(), t);
        ^
/usr/include/boost/archive/basic_binary_iarchive.hpp:70:37: note: in instantiation of function template specialization
      'boost::archive::detail::common_iarchive<boost::archive::binary_iarchive>::load_override<navitia::type::new_disruption::Tag>' requested here
      this->detail_common_iarchive::load_override(t, static_cast<int>(version));
                                    ^
/usr/include/boost/archive/binary_iarchive_impl.hpp:50:47: note: in instantiation of function template specialization
      'boost::archive::basic_binary_iarchive<boost::archive::binary_iarchive>::load_override<navitia::type::new_disruption::Tag>' requested here
        this->basic_binary_iarchive<Archive>::load_override(t, 0L);
                                              ^
/usr/include/boost/archive/detail/interface_iarchive.hpp:60:23: note: in instantiation of function template specialization 'boost::archive::binary_iarchive_impl<boost::archive::binary_iarchive, char, std::char_traits<char>
      >::load_override<navitia::type::new_disruption::Tag>' requested here
        this->This()->load_override(t, 0);
                      ^
/usr/include/boost/serialization/nvp.hpp:87:12: note: in instantiation of function template specialization 'boost::archive::detail::interface_iarchive<boost::archive::binary_iarchive>::operator>><navitia::type::new_disruption::Tag>'
      requested here
        ar.operator>>(value());
           ^
/usr/include/boost/serialization/access.hpp:101:11: note: in instantiation of function template specialization 'boost::serialization::nvp<navitia::type::new_disruption::Tag>::load<boost::archive::binary_iarchive>' requested here
        t.load(ar, file_version);
          ^
/usr/include/boost/serialization/split_member.hpp:54:21: note: (skipping 247 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
            access::member_load(ar, t, file_version);
                    ^
/usr/include/boost/serialization/export.hpp:66:12: note: in instantiation of member function 'boost::serialization::singleton<boost::archive::detail::pointer_iserializer<boost::archive::binary_iarchive,
      navitia::type::DiscreteVehicleJourney> >::get_const_instance' requested here
        >::get_const_instance();
           ^
/usr/include/boost/serialization/export.hpp:111:40: note: in instantiation of member function 'boost::archive::detail::export_impl<boost::archive::binary_iarchive, navitia::type::DiscreteVehicleJourney>::enable_load' requested here
    export_impl<Archive,Serializable>::enable_load(
                                       ^
/usr/include/boost/serialization/export.hpp:95:37: note: in instantiation of member function 'boost::archive::detail::ptr_serialization_support<boost::archive::binary_iarchive, navitia::type::DiscreteVehicleJourney>::instantiate' requested
      here
        &ptr_serialization_support::instantiate
                                    ^
/usr/include/boost/serialization/export.hpp:142:9: note: in instantiation of member function 'boost::archive::detail::extra_detail::guid_initializer<navitia::type::DiscreteVehicleJourney>::export_guid' requested here
        export_guid(boost::serialization::is_abstract< T >());
        ^
/home/fred/navitia/source/type/type.cpp:772:1: note: in instantiation of member function 'boost::archive::detail::extra_detail::guid_initializer<navitia::type::DiscreteVehicleJourney>::export_guid' requested here
BOOST_CLASS_EXPORT_GUID(navitia::type::DiscreteVehicleJourney, "DiscreteVehicleJourney")
^
/usr/include/boost/serialization/export.hpp:189:72: note: expanded from macro 'BOOST_CLASS_EXPORT_GUID'
BOOST_CLASS_EXPORT_KEY2(T, K)                                          \
                                                                       ^
/usr/include/boost/serialization/export.hpp:167:35: note: expanded from macro '\
BOOST_CLASS_EXPORT_IMPLEMENT'
        >::get_mutable_instance().export_guid();             \
                                  ^
1 warning and 1 error generated.

[antoine-de]

It's very strange, some people in the team use Clang as their main compiler 😕

What I don't understand is that it stops at the template depth limit of 256, where the limit is explicitly set to 1024 in the CMakeList (https://github.com/CanalTP/navitia/blob/dev/source/CMakeLists.txt#L29).

[frodrigo]

By cleaning completely my release directory and not just make clean it, it's compile with clang from gcc.

osm2ed compiled by clang die with the same error.

[pbougue]

So the only lead I have left is debug + gdb, as we cannot reproduce it here... :-/

[eturck]

Hello!

While trying to test a few things on Kraken I realize I haven't yet imported OSM datas.
I got the same exact error when I import the Toulouse area. I tried with different set of datas and got the exact same error each time.

GDB gave me the backtrace, the failing assert was happening : https://github.com/CanalTP/navitia/blob/cc9eab91c5e77adc7983aa80a6957d9495235873/source/ed/osm2ed.cpp#L197

    for (const auto& relation : relations) {
        relation.build_geometry(*this);
        boost::geometry::model::box<point> box;
        boost::geometry::envelope(relation.polygon, box);
        Rect r(box.min_corner().get<0>(), box.min_corner().get<1>(),
                box.max_corner().get<0>(), box.max_corner().get<1>());
CRASH > admin_tree.Insert(r.min, r.max, &relation); < CRASH
    }

I used 2 differents sets of datas, one from geofabrik, one from mapzen metro-extracts and got 2 other sets by clipping them based on the polygon of our working area.
Each time, the relation when the assert happened was one at the border of the OSM datas.
With mapzen datas it was :

And the relation was : http://www.openstreetmap.org/relation/135621

The polygon returned by relation.build_geometry was empty, but it wasn't the problem since many polygons before this one were empty and successfully added to the admin_tree.
I don't really understand everything in Rtree so I tried another approach. Since the admin_tree is (I think) only used in match_coord_admin to find which admin is associated with a set of coords.
I just added this :

    for (const auto& relation : relations) {
        relation.build_geometry(*this);
+       if(relation.polygon.empty())
+           continue;
        boost::geometry::model::box<point> box;
        boost::geometry::envelope(relation.polygon, box);
        Rect r(box.min_corner().get<0>(), box.min_corner().get<1>(),
                box.max_corner().get<0>(), box.max_corner().get<1>());
        admin_tree.Insert(r.min, r.max, &relation); 
    }

And everything seems alright =).
I don't know if there is some side effects, but since we're ignoring admins with empty polygon at the end I guess it's not really bad? ( https://github.com/CanalTP/navitia/blob/cc9eab91c5e77adc7983aa80a6957d9495235873/source/ed/osm2ed.cpp#L380 )

Let me know if you need any precision of if you want me to test something else. It would probably be better to really fix the problem but I don't think I understand the Rtree enough to really find the right fix to this...

[antoine-de]

Hi @eturck , the fix seems right since the empty relation does not seems to be useful.

I finally managed to reproduce the crash, there seems to be an useless assert in the rtree library. Since we never use the debug compiled osm2ed we never saw this 😕

I check the result between the patched version and the previous one, and apart from a problem with the algorithm that merge the ways (patched in the same PR 😉 ), it seems similar.

Thanks @eturck for this great analysis !

By the way, why do you use debug compiled executables @eturck, @frodrigo ? There can be an order of magnitude on the compilation duration and on the execution time (especially for the part where boost::serialize are used).

[eturck]

Thanks @antoine-de! 🙏

As for the debug part... Mainly because on the server I was we didn't bother to specify anything to cmake...
I thought CMAKE_BUILD_TYPE would be release by default and I never checked. (oops https://github.com/CanalTP/navitia/blob/dev/source/CMakeLists.txt#L23 )

Thanks for the reminder ;)
(I told you navitia was so fast already we didn't see anything wrong!)

[antoine-de]

Oups I though it was the Cmake default 😉

I'm going to change this, debug is soooo slow.

Thanks!

On Fri, Oct 9, 2015 at 9:43 AM, Erwan [email protected] wrote:

Thanks @antoine-de https://github.com/antoine-de! [image: 🙏]

As for the debug part... Mainly because on the server I was we didn't
bother to specify anything to cmake...
I thought CMAKE_BUILD_TYPE would be release by default and I never
checked. (oops
https://github.com/CanalTP/navitia/blob/dev/source/CMakeLists.txt#L23 )

Thanks for the reminder ;)
(I told you navitia was so fast already we didn't see anything wrong!)

—
Reply to this email directly or view it on GitHub
#1115 (comment).