Appveyor config about libosmium

[alex85k]

It seems that CMake and nuget packages do not like each other :)
Package managers on Windows (not for .NET) are very weak. (MSYS2 have a good one but it is not MSVC-based).

nuget tries to add packages directly into solution config but cmake uses its custom library search and highly customized solution files...

For example, the path for expat in nuget is like expat.2.1.0.11\build\native\lib\v110\x64\Release\static\utf16\libexpatw.lib . I doubt CMake will ever find it without hardcoded paths (and VS version changed too).
I also could not find any .lib files in the package protobuf (maybe it is broken or separated into pieces).

I would suggest to download your custom-build dependencies pack with classic layout (bin / lib / include) for building libosmium on appveyor, like it is done in OSRM:
https://github.com/Project-OSRM/osrm-backend/blob/master/appveyor.yml#L29

This is not the best solution but it will work.

[joto]

Well it means somebody has to create that package compiling all the dependencies etc. I can't do that, because I don't have a Windows system. So it would hinge around somebody else doing that and re-doing it again and again when something changes, new versions come out etc. That doesn't sound like a long-term solution.
Is nuget the only game in town or is there some other, better, packaging solution?
I am sure we can create a script that finds the libs in the nuget packages and moves them in a place where cmake can find them or do that in cmake somehow if needed.

[alex85k]

The best variant for all Mapbox-related projects could be auto-packaging and updating dependencies
For Mapnik it is in progress, I guess: https://github.com/BergWerkGIS/mapnik-dependencies (@BergWerkGIS and @springmeyer), all the needed libs are there.

Nuget is only packaging solution for MSVC that I know, mostly used for .NET projects. There is also http://trac.osgeo.org/osgeo4w/ but it updates very rarely.

Theoretically, we can try to do a huge configuring/renaming to force CMake use scattered libraries and headers from nuget folders, but maybe some packages will not work due to MSVC version changes (yes, it is that bad - libraries from older and newer MSVC are often incompatible). I have found a package with protobuf libs:

Package: protobuf and protobuf-vc110
Lib: protobuf-vc110.2.5.0.0\build\native\lib\x64\v110\Release\libprotobuf.lib
Includes: protobuf.2.5.0.0\build\native\include

Package: expat
Lib: expat.2.1.0.11\build\native\lib\v110\x64\Release\static\utf8\libexpat.lib
Inlcude: expat.2.1.0.11\build\native\include

but this is to be continued and checked, which can take a lot of time that I do not have now...

[alex85k]

Package: bzip2
Library: bzip2.1.0.6.11\build\native\lib\v110\x64\Release\static\stdcall\libbz2.lib (most likely, maybe cdecl instead of stdcall)
Include: bzip2.1.0.6.11\build\native\include

Package: zlib
Library: zlib.1.2.8.7\build\native\lib\v120\x64\Release\static\stdcall (most likely, maybe cdecl instead of stdcall)
Include: zlib.1.2.8.7\build\native\include

Package: boost
Include: boost.1.57.0.0\lib\native\include

Package: gdal
Nuget is not suitable for C/C++ gdal development: no header and .lib files in packages, only .dlls used by .NET wrapper...

OSMPBF and wingetopt should be built or downloaded manually (no nuget packages).
So using nuget seems to be very complex task.

[springmeyer]

Yes, would make sense to extend the mapnik-dependencies scripts to help address the easy installation of pre-built dependencies for libosmium. @BergWerkGIS and I are planning is to get https://github.com/mapbox/mason going on windows soon, which is the successor to https://github.com/BergWerkGIS/mapnik-dependencies, so that would be the ideal solution when ready.

[wilhelmberg]

@joto any idea what's wrong?

I did:

git clone https://github.com/osmcode/libosmium.git
cd libosmium
mkdir build
cd build

cmake .. ^
-DBOOST_ROOT=%PKGDIR%\boost ^
-DOSMPBF_LIBRARY=C:\mb\mapnik-dependencies-64\packages\OSM-binary\deploy\lib\osmpbf.lib ^
-DOSMPBF_INCLUDE_DIR=C:\mb\mapnik-dependencies-64\packages\OSM-binary\deploy\include ^
-DPROTOBUF_LIBRARY=%PKGDIR%\protobuf\vsprojects\%BUILDPLATFORM%\%BUILD_TYPE%\libprotobuf-lite.lib ^
-DPROTOBUF_INCLUDE_DIR=%PKGDIR%\protobuf\src ^
-DZLIB_LIBRARY=C:\mb\mapnik-dependencies-64\packages\zlib\zlib.lib ^
-DZLIB_INCLUDE_DIR=C:\mb\mapnik-dependencies-64\packages\zlib ^
-DEXPAT_LIBRARY=C:\mb\mapnik-dependencies-64\packages\expat\win32\bin\Release\libexpat.lib ^
-DEXPAT_INCLUDE_DIR=C:\mb\mapnik-dependencies-64\packages\expat\lib ^
-DBZIP2_LIBRARIES=C:\mb\mapnik-dependencies-64\packages\bzip2\libbz2.lib ^
-DBZIP2_INCLUDE_DIR=C:\mb\mapnik-dependencies-64\packages\bzip2

[springmeyer]

@BergWerkGIS - great progress! As far as the error, maybe try forward slashes or double backslash for path separators? I don't know Cmake much but googling I found http://stackoverflow.com/a/13052993/2333354

[wilhelmberg]

@springmeyer thanks. Yes, forward slashes solve this. Already discussed with @joto on chat.

Problem is, we use backward slashes in all our env vars.

Solved it with a string replace (\ -> /) just for the bzip2.lib path.

[wilhelmberg]

Seems, I got all deps working.

Current problem:
Conflicts between zlib and gdal.
I think, because we build gdal with tiff, which we build with zlib.

@joto I suppose libosmium doesn't need tiff support?
Should be enough to create a gdal build without tiff?

[joto]

@BergWerkGIS Osmium itself doesn't need Tiff support, but at least one of the examples (node_density in https://github.com/osmcode/osmium-contrib) does. So eventually it would be good to solve this, but not highest priority.

Lets get what we have into the appveyor config and see where we are then.

[springmeyer]

@BergWerkGIS - looks like gdal may be statically linking zlib? Perhaps the problem is we need to dynamically link gdal to both libtiff and zlib (and this may fix mapbox/mason#7 at the same time?)

[wilhelmberg]

@joto solved the zlib issue by linking to gdal_i.lib instead of gdal.lib.

But I have a new error (target: geom_test_factory_with_projection):

C:\mapbox\mapnik-dependencies-32\packages\libosmium\include\osmium/util/cast.hpp(58): error C2338: unnecessary static_cast_with_assert when casting into type of equal or larger size

[joto]

@BergWerkGIS Are you are using a 32bit system? 32bit are not fully supported, see #56.

[wilhelmberg]

@joto you were right, 64bit did get me past that error.
Afterwards I had to move some header files around for gdal and sparsehash.
Now I'm stuck at a new zlib error.
Might have something to do that zlib gets built as part of libpng.
Will investigate further and try a dedicated build of zlib.

[wilhelmberg]

@joto
A dedicated zlib build and some other changes got me to 82%.

Now I get error C2280: attempting to reference a deleted function.
Any ideas?

[wilhelmberg]

Full output for above mentioned error:
https://gist.github.com/BergWerkGIS/007c0bc8938f5734323b

[wilhelmberg]

Just for the record: commenting project osmium_convert made the process run through to 100%.

[wilhelmberg]

Latest build error that stopped the process at 1% was due to using -DCMAKE_BUILD_TYPE=Dev with sets flag Werror here.
It seems that this is not available on Windows.

[wilhelmberg]

@joto
YES! 🚀
Finally AppVeyor is working, but the attempting to reference a deleted function error still persists.

Should I issue a PR with the changes to appveyor.yml?

[joto]

Everything compiles now on Appveyor. In addition Appveyor is now set up so that Release and Dev builds are both checked. And I have added a call to ctest. The tests don't run all through. I will open extra issues for those cases.