Buffer::purge_removed() crashes while using MultipolygonCollector about libosmium

[joto]

@fbielig can you verify the bug is gone?

This bug is really gone. Thank you a lot for fast reactions.

Unfortunately my program stops with another segmentation fault. Stack trace you can see below. in generell I do not more than using your multipolygon assembler with tens of thousends of features. Probably you can reproduce my issue by dumping all multipolygons of a larger area. For small data sets everythings works pretty well.

---8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<---
0 osmium::memory::Item::byte_size item.hpp 150 0x5a5cf2
1 osmium::memory::Collection<osmium::Tag, (osmium::item_type)17>::cend collection.hpp 136 0x5bb1a4
2 osmium::memory::Collection<osmium::Tag, (osmium::item_type)17>::end collection.hpp 144 0x5bb148
3 osmium::area::Assembler::operator() assembler.hpp 746 0x6016fa
4 osmium::area::MultipolygonCollectorosmium::area::Assembler::complete_relation multipolygon_collector.hpp 166 0x61f8be
5 osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2::find_and_add_object collector.hpp 175 0x61dc06
6 osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2::way collector.hpp 201 0x61bf43
7 osmium::detail::apply_item_recurseosmium::relations::Collector<osmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2, osmium::memory::Item> visitor.hpp 72 0x6197b5
8 osmium::detail::apply_item_recurse<m1::osm::osmium::clipping::ClipCropped::Pass2, osmium::memory::Item, osmium::relations::Collector<osmium::area::MultipolygonCollectorosmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 209 0x614f15
9 osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>, m1::osm::osmium::clipping::ClipCropped::Pass2, osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 238 0x60e8d4
10 osmium::apply<osmium::io::Reader, m1::osm::osmium::clipping::ClipCropped::Pass2, osmium::relations::Collector<osmium::area::MultipolygonCollectorosmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 245 0x608714
11 m1::osm::osmium::clipping::ClipCropped::clip clip_cropped.cpp 962 0x5f55fc
12 main m1osm_clipper.cpp 312 0x58bb4b

[joto]

For me it runs fine with, for instance, Germany. Don't know whether that counts as a "larger dataset" for you. :-)

It is a bit difficult to debug without access to the code and an example file. Can you reproduce this with one of the example programs and tell me which data file you are using?

Are you sure you are not simply running out of memory?

Currently I'm working on a planet extract of South America. Unfortunately my issue is not reproducable with your examples. I'm still trying to nail this bug down. Even GDB crashes while debugging. Memory outage is not the problem.

[alex85k]

It is certainly the the same bug that broke the testing in #42 (comment) : iterator in find_and_add_object becomes undefined after "erase" call in complete_relation method. Sometimes it works fine, sometimes give an assertion (with strictly-iterator-checking Debug MS STL), sometimes crashes :)

@fbielig : you can try small example https://raw.githubusercontent.com/osmcode/osm-testdata/master/grid/data/all.osm . Maybe it will crash too (it does on Windows at #42 )

Now I have a data file generating a failing assertion. This time I used test program libosmium_area_test. My test file still has nearly 40MB. I try to get it smaller and send it to @joto directly. Is it ok?

3 __GI___assert_fail 101 0x7ffff6bc2b22
4 osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2::find_and_add_object collector.hpp 166 0x545f64
5 osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2::way collector.hpp 201 0x542fad
6 osmium::detail::apply_item_recurseosmium::relations::Collector<osmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2, osmium::memory::Item> visitor.hpp 72 0x53e5ae
7 osmium::detail::apply_item_recurse<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location>>, osmium::memory::Item, osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 209 0x538145
8 osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>, osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location>>, osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 238 0x52fb12
9 osmium::apply<osmium::io::Reader, osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location>>, osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 245 0x525b58
10 main osmium_area_test.cpp 120 0x50117f

@alex85k: unfortunatly your small example works on my side.

[alex85k]

@fbielig : I am almost sure the both fails will be fixed together when @joto will find some time to investigate (it is not very easy to fix, so I can not make a patch). The reason from previous comment is most likely (99%) correct :)

[joto]

Hopefully fixed in 9edb40d.

Thank you for you efforts. This issue is really fixed in my test cases.

PS: Next issue will come soon ;)

I will send you a new test dateset creating a segfault in libosmium_area_test. Probably it relates to the quite complex multipolygon 273387779.