cmake test build doesn't build stuff in osm-testdata about libosmium

[alex85k]

To compile the tests we need usual {} initialization patch: alex85k@2218602

The unfinished cmake script is here: alex85k@6744dc3
(cloning osm-testdata included). For now I can not build GDAL with Spatialite to be able to run tests.
To compare logs we'll also need installed ruby (not a problem on Windows too).

[alex85k]

Here is the version that runs all tests
https://github.com/alex85k/libosmium/commits/cmake-osmtestdata
(without comparing areas for now)

[alex85k]

I have rebased the patches to current libosmium version and tried to build on Visual Studio 14 CTP.

The crashes during tests remain, so there are still some serious problems with Osmium on Windows.

I have created the "buggy" temporary test: alex85k@0ae2b93

The unspecied exception usually stops the program at https://github.com/osmcode/libosmium/blob/master/include/osmium/thread/checked_task.hpp#L78 and close() function. m_future.get crashes while rethrowing the parsing exception, but not every time (!), so this is most likely memory error related to packaged_task or m_future lifecycle.

[joto]

I have added your commit eedfcfa with the cmake support for the osm-testdata test and a little script to run the multipolygon test properly. The script will most likely not work on Windows (it uses the shell), but I don't know how else to fix it.

[joto]

I had problems similar to what you describe on Linux, but got them fixed. Difficult to debug without access to a Windows system...

[alex85k]

Thank you for including the scripts. Could you also include alex85k@bb02e18 to fix compilation errors?

I'll try to debug once more... Often the error was appearing by close() calls in the destructors.
(like https://github.com/osmcode/libosmium/blob/master/include/osmium/io/detail/xml_input_format.hpp#L685 ).

[joto]

I have added alex85k/libosmium@bb02e18.

[alex85k]

It seems I have found a problematic place and possible fix. Currently the crashes are caused by incorrect processing of exceptions in the destructor. Maybe something like this:
http://stackoverflow.com/questions/130117/throwing-exceptions-out-of-a-destructor

So I noticed the exception-eating destructor in your code and applied the same safeguard to ~Reader() destructor. The bugs stopped appearing, the tests did not crash!
Here it is: alex85k@98355e4
(do not know about the correct logic).

There seems to be few missing files in osm-testdata (do not know how to fix).
Comparator script is needed to be re-implemented for Windows (will do it later):
https://gist.github.com/alex85k/a8598a6ee2e7df914b06

[alex85k]

When the multipolygon test is re-enabled, it still crashes on iterator checking:
http://i.imgur.com/4t7fAe6.png

It may be the same bug as #47 because osmium::relations::Collector is involved.

[alex85k]

Patch to enable testdata-multipolygon on Windows: alex85k@a72409e (exe is still crashing, needs the author's attention :) )

[joto]

Files were missing becaus I forgot to commit/push them. Now done.
I also merged your Reader destructor patch. I added explicit close() calls to all tests and examples, so if there are exceptions they should not be ignored this way.
And I merged your multipolygon test .bat.
Still need to find the Collector bug(s).

[alex85k]

Thank you! I confirm that the changes are working on Windows.
testdata-xml passed successfully!

[alex85k]

By the way, don't forget to add new examples into cmakelists.txt :)
alex85k@ac40317

[joto]

I have pushed a fix for #47 . Can you check whether the multipolygon test now works?

[alex85k]

New test_buffer_purge passes (when added into https://github.com/osmcode/libosmium/blob/master/test/CMakeLists.txt#L54 ), but multipolygon test still crashes with the same exception exception2...

[alex85k]

And iе seems to be caused by possibly_purge_removed_members ... (it somehow makes it++ give an error on next step...)

[joto]

Hm. That looks like the bug I just fixed. You sure you got the current code?

[alex85k]

Hm. That looks like the bug I just fixed. You sure you got the current code?

I am afraid so, the code is latest :) http://i.imgur.com/jxpPGHz.png

[alex85k]

But is seems I am wrong, the error can be caused by m_collector.complete_relation(relation_meta);, not possibly_purge_removed_members();.

Commenting possibly_purge_removed_members() did not remove the crash, but commenting m_collector.complete_relation(relation_meta) "solved" the problem (the results are most likely incorrect but there was no crash),

[alex85k]

Another hint: commenting the only mmv.erase(it) line in https://github.com/osmcode/libosmium/blob/master/include/osmium/area/multipolygon_collector.hpp#L187 removed the crash (in my small experience erase is the most problematic STL method...)

[alex85k]

Maybe this is the case?
http://stackoverflow.com/questions/2874441/deleting-elements-from-stl-set-while-iterating

But the proposed erase(it++) fix did not help...
P.S. However, the break makes this iterator unnecessary for future use.
So there is maybe some logical error that does not show up with full-length vector (without erasing).

[alex85k]

After debbugging: range.first variable is shifted when mmv.erase(it) is done in complete_relation(). It is shifted not only in complete_relation() (may be ignored), but in calling function too:
https://github.com/osmcode/libosmium/blob/master/include/osmium/relations/collector.hpp#L163

[alex85k]

More important, it iterator is changed during mmv.erase(it) call in different fucntion.
Even after adding auto tmp = it; into the loop, tmp is automatically changed when erasing.
Something like http://stackoverflow.com/questions/6438086/iterator-invalidation-rules

Some logic change is needed, I am afraid. Tricks like described will not work with such implicit erasing. We sometimes delete the element, so we can not iterate over it...
P.S. I can not fix this myself :)

[alex85k]

cmake patch for two new tests: alex85k@92aa327

[joto]

Thanks for the analysis @alex85k and thanks for the test file @fbielig. This made it much easier to fix. Please validate that the fix works.

[alex85k]

Thank you very much, this seems to be fixed!

100% tests passed, 0 tests failed out of 36

Total Test time (real) =  10.91 sec

I'll check why the test pass without Ruby installed later 😄 (but there is certainly no crash - tested multiple times).

[alex85k]

After patching scripts: test fails because Ruby uses unsupported compare-wkt.sh which use unsupported spatialite executable (this can be found if needed)... It would be great if you can integrate compare-wkt.sh into ruby script (should be easy).

[joto]

Don't know what to do about that. I used existing software as much as possible for comparing test results instead of writing my own software, because that would need tests, too... Maybe somebody else has a clever idea how to do those tests in a more portable way. Until then I suggest we just disable those parts of the test on Windows.

[alex85k]

For now it "works" as needed - counts the test as "passed". Later I can try to integrate compare-wkt as ruby module.

[joto]

Okay, I'll close this issue now and open a brand new one documenting that we still have to finish that test somehow.