Windows support about libosmium

[springmeyer]

Great to see this coming together!!

[DennisOSRM]

👍

[alex85k]

@joto Wow! Your speed and organization talents are amazing :)

[alex85k]

The try-to-fix branch v2:
https://github.com/alex85k/libosmium/commits/msvc-v2
(added snprintf issue: alex85k@2a9f59a )
Last (?) compilation errors are here https://gist.github.com/alex85k/cf68015851f1b3c2f5ef

• Some mmap support for Windows like alex85k/osmium@e486c49 is to be added
• some remaining errors from #32
• later I'll add tests compile&run to cmake, maybe some more issues will show up :)

[alex85k]

Branch v3 (with enabled static_asserts):
https://github.com/alex85k/libosmium/tree/msvc-v3

[alex85k]

After fixing #32 and temporary disabling changeset sizeof checking, only mmap and linker issues persist. Preparing a buildable branch...

[joto]

@alex85k: have added your mmap code from old Osmium in dcb8902. Please check whether this works.

[alex85k]

Just did it too :) Rebasing and removing that patch...
https://github.com/alex85k/libosmium/commits/msvc-v4

The examples build now! (Not sure if they work correctly)

[DennisOSRM]

Looking forward to plug this into OSRM tomorrow

[alex85k]

The remaining patches:
https://github.com/alex85k/libosmium/commits/msvc-v5
(examples build with build.bat, no more errors)

Please rework alex85k@6e430a1

By the way, alex85k@aba2668 from checklist was integrated into first cmake patch together with other options.

[alex85k]

I am adding tests build&run to cmake (like for old osmium) and checking build on mingw.

[alex85k]

area and basic tests are in https://github.com/alex85k/libosmium/commits/msvc-v5
However, there are 2 errors:

• there is an assertion in https://github.com/osmcode/libosmium/blob/master/test/t/basic/test_location.cpp#L10 (not literal type)
• there is a huge compile error in nullptr checking at https://github.com/osmcode/libosmium/blob/master/test/t/basic/test_node.cpp#L104 :
  https://gist.github.com/alex85k/cf68015851f1b3c2f5ef#file-log-txt

[alex85k]

Only 11 tests of all does not compile (sometimes incompatibilities in test code, sometime missing libraries solvable by cmake), they are commented in https://github.com/alex85k/libosmium/blob/msvc-v5/test/CMakeLists.txt

[alex85k]

For example, geos tests do not compile because of unsupported std::throw_with_nested in https://github.com/alex85k/libosmium/blob/msvc-v5/include/osmium/geom/geos.hpp#L97
In other tests there are also errors like

..\..\test\t\tags\test_filter.cpp(37): error C2668: 'osmium::builder::build_tag_list' : ambiguous call to overloaded function [D:\build_d\libosmium4\build-cmake\test\test_filter.vcxproj]
..\..\test\t\tags\test_tag_list.cpp(16): error C2668: 'osmium::builder::build_tag_list' : ambiguous call to overloaded function [D:\build_d\libosmium4\build-cmake\test\test_tag_list.vcxproj]
..\..\test\t\index\test_id_to_location.cpp(9): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory [D:\build_d\libosmium4\build-cmake\test\test_id_to_location.vcxproj]
..\..\test\t\index\test_typed_mmap.cpp(72): error C3861: 'mkstemp': identifier not found [D:\build_d\libosmium4\build-cmake\test\test_typed_mmap.vcxproj]
C:\Program Files (x86)\Microsoft Visual C++ Compiler Nov 2013 CTP\include\utility(53): error C2678: binary '=' : no operator found which takes a left-hand operand of type 'const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>' (or there is no acceptable conversion) [D:\build_d\libosmium4\build-cmake\test\test_wkt.vcxproj]

Good news is that other tests pass without failures (except Bzip-one, file-related)

I am afraid to think what errors we may encounter in hpps not covered by tests :D

[joto]

The assertion in test_location.cpp isn't important. I have removed it in 10bc9fc.
The nullptr compile error probably comes from a missing overload of operator<<(ostream, nullptr_t). I can't find such an overload on Linux either, so maybe I am wrong. You can test by adding

template <typename TChar, typename TTraits>
inline std::basic_ostream<TChar, TTraits>& operator<<(std::basic_ostream<TChar, TTraits>& out, const nullptr_t) { return out << '0'; }

to the test file and see whether it fixes it.

Also see http://cplusplus.github.io/LWG/lwg-active.html#2221 and catchorg/Catch2#80

[alex85k]

Ok, will try it tomorrow.

[joto]

You can also try defining 'CATCH_CONFIG_CPP11_NULLPTR' first thing before including catch.hpp.

[alex85k]

You can also try defining 'CATCH_CONFIG_CPP11_NULLPTR' first thing before including catch.hpp.

Thank you, this helped!
(when added to both test_main.hpp and test_node.hpp).

[alex85k]

All headers are compiling successfully now (added BUILD_TRY_HPPS option to cmake)!

All tests are activated, 7 of them did not build for now. BZIP test failure was solved by setting work directory.

Here are the commits that may be still needed:
https://github.com/alex85k/libosmium/commits/msvc-v6

[alex85k]

Considering checklist:
alex85k@09511be was replaced by alex85k@4de4e18
, alex85k@d07f43e just included as 72844f4

[joto]

@alex85k: I did a hack for the CATCH_CONFIG_CPP11_NULLPTR issue in 12bf8ca. This really has to be integrated upstream (https://github.com/philsquared/Catch). catch.hpp checks for compiler versions to see whether it should set this. Could you figure out which version of MSVC has nullptr and submit a patch to upstream? Or at least open an issue?

[alex85k]

nullptr is available since Visual Studio 2010. I'll try to submit a patch.

[joto]

@alex85k: I have merged your cmake files and made quite a few changes. Still work in progress. See also the cmake config in https://github.com/osmcode/osmium-tool which I added a while back. I think I want to keep the old Makefile around for those people who understand Makefiles and not cmake. Also there are currently a lot more targets that I need and they are more convenient to use than with cmake. Somehow I have to keep Makefiles and CMakeLists in sync though. We'll see. Any help appreciated, I am still quite new at this cmake stuff.

[alex85k]

Thank you very much for your work!

I agree that makefile will be useful too. It is certainly simpler and the only drawback is "exotic" environments like Windows and poorer IDE support. Most of the complex libraries have 2 or more build systems for different cases. I'll try to check cmake version periodically and also try to help with continuos integration on Windows (appveyor.com is like travis) to automatically detect incompatibilities. By the way, additional targets can also be added to cmake later, if really needed (yes, it supports arbitrary shell commands).

My first experience with cmake was exactly for building TagInfo based on Osmium (it was a real pain to find all that necessary libraries "by hand", so I just copied fragments of other projects scripts at the beginning), so it seems rather easy (especially compared to autotools). I liked it so much that even tried to push it into osmpbf, protobuf-c and osm2pgsql :)

Now we have only few win-bugs remaining (if it works correctly with real files after all...).

[alex85k]

Here is the pull-request for Catch: catchorg/Catch2#307

[alex85k]

Remaining issues:

• #37
• #38
• alex85k@c71b03f (trivial, the project was renamed)
• #39
• already mentioned #36 and alex85k@2d039c3

[alex85k]

After adding the last patch alex85k@5931171 for #38 I have all the tests building and passing successfully on Windows!

[DennisOSRM]

Yay!

[joto]

Awesome!

@alex85k: If you want to you can continue with https://github.com/osmcode/osmium-tool and https://github.com/osmcode/osmium-contrib. :-)

[alex85k]

@joto: I have compiled GDAL and added the tests to cmake:
alex85k@59b6ae7
I also tried to make it more robust if some libraries are not found - could you look?

My build scripts are now collected in https://github.com/alex85k/winbuilds

[joto]

I have merged your cmake changes (and fixed the indentation :-).

I'll open some extra cmake-related issues.

[alex85k]

Windows compiltation is broken again after adding <endian.h> in 01831cc
We need something like

#ifdef _WIN32
#ifndef __BYTE_ORDER
  #define __LITTLE_ENDIAN 1234
  #define __BIG_ENDIAN    4321
  #define __BYTE_ORDER __LITTLE_ENDIAN
#endif
#else
  #include <endian.h>
#endif

[joto]

Is that safe? Does Windows only run on little endian maschines?

[alex85k]

Yes, all "normal" Windows versions are for little-endian x86 .
http://stackoverflow.com/questions/6449468/can-i-safely-assume-that-windows-installations-will-always-be-little-endian
(however, the kernel is universal)
We can assume for now that no one will run osmium tools on hacked XBox 360 :)

[joto]

Fixed in 77f59d2.

[alex85k]

There is also one more build.bat fix: alex85k@dc02922
(my mistake wuth debug/release switchng)

[alex85k]

Could you please also remove <endian.h> from test_wkb.cpp ? It is already included by wkb.hpp.

[springmeyer]

Want to note I've been working on getting Mapnik 3.x (first release to use c++11) + all deps built with both the 2013 Nov CTP and the 2014 CTP 3. Details at mapnik/mapnik#2396 and scripts at https://github.com/BergWerkGIS/mapnik-dependencies/tree/windows.

Overall I've managed to get Mapnik compiled and linked with both compilers but am seeing mysterious hangs at runtime with the 2013 Nov CTP. Because the 2014 CTP 3 supports more C++11 features (namely better constexpr support) and I'm not seeing program hangs with it I've given up on using the 2013 CTP. The challenge with the 2014 CTP is that commits like alex85k@2a9f59a will break because C99 stdio functions like snprintf were actually finally implemented. So I've needed to patch around these issues like https://github.com/BergWerkGIS/mapnik-dependencies/blob/windows/patches/libxml2.diff or https://github.com/BergWerkGIS/mapnik-dependencies/blob/windows/patches/libtiff.diff.

So, before the work here settles I would recommend taking a serious look at dropping support for the 2013 CTP and instead preferring the 2014 CTP because getting things working with the 2014 CTP will likely be less work in the long run.

[alex85k]

@springmeyer :Thank you very much for your experience! It seems I was unwillingly repeating the dependency build scripts (simpler variant) at https://github.com/alex85k/winbuilds (just finished my painful GDAL+plugins like spatialite compilation). You can take and adapt GEOS+cmake from it, if needed (however, makefile.vc is surely enough) and I'll take yours when needed :)

I also encountered strange hangs on VS 2013 CTP (osmium tool does it for pbf).

[alex85k]

@joto: Good news: the remaining tests are compiling and running.
Bad news: there are memory errors detected and "abort" exceptions arise in testdata-xml and testdata-multipolygon. They are triggered not every time (after commenting many tests to detect failing one I got 50% of crashing).

The Intel Inspector XE says that there are memory errors in m_parser_task(input_queue, m_queue, m_header_promise, read_which_entities, m_done) (something like copying atomic from non-initialized memory). So maybe that XMLParser copying in CheckedTask still contains some problems...

[alex85k]

This test is causing a crash in 30% of runs, for example:

TEST_CASE("Reading OSM XML 103") {
   SECTION("Using Reader") {
        REQUIRE_THROWS_AS({
            osmium::io::Reader reader(filename("103-old_version"));
            osmium::io::Header header = reader.header();
            osmium::memory::Buffer buffer = reader.read();
            reader.close();
        }, osmium::format_version_error);
    }

}

[joto]

@alex85k has this all been resolved? Can we close this issue?

[alex85k]

Yes, I guess so. Thank you!

[alex85k]

It seems I have found a thread problem that makes the osmium_tool hang on Windows: #49

[joto]

Okay #49 is resolved, I close this now. We can always open more issues if needed...

[alex85k]

Osmium-contrib building script and compatibilty changes are ready: osmcode/osmium-contrib#2