osmcode / libosmium

Fast and flexible C++ library for working with OpenStreetMap data.

Libosmium

https://osmcode.org/libosmium

A fast and flexible C++ library for working with OpenStreetMap data.

Libosmium works on Linux, macOS and Windows.

Please see the Libosmium manual for more details than this README can provide.

Prerequisites

You need a C++11 compiler and standard C++ library. Osmium needs at least GCC 4.8 or clang (LLVM) 3.4. (Some parts may work with older versions.)

Different parts of Libosmium (and the applications built on top of it) need different libraries. You DO NOT NEED to install all of them, just install those you need for your programs.

For details see the list of dependencies in the manual.

The following external (header-only) libraries are included in the libosmium repository:

• gdalcpp

Note that protozero was included in earlier versions of libosmium, but isn't any more.

Directories

• benchmarks: Some benchmarks checking different parts of Libosmium.

• cmake: CMake configuration scripts.

• doc: Config for API reference documentation.

• examples: Osmium example applications.

• include: C/C++ include files. All of Libosmium is in those header files which are needed for building Osmium applications.

• test: Tests (see below).

Building

Osmium is a header-only library, so there is nothing to build for the library itself.

But there are some tests and examples that can be build. Libosmium uses cmake:

mkdir build
cd build
cmake ..
make

This will build the examples and tests. Call ctest to run the tests.

For more details see the Building Libosmium chapter in the manual.

Testing

To download the osm-testdata submodule call:

git submodule update --init

This will enable additional tests.

See the Libosmium Manual for instructions.

License

Libosmium is available under the Boost Software License. See LICENSE.

Authors

Libosmium was mainly written and is maintained by Jochen Topf ([email protected]). See the git commit log for other authors.

libosmium's Issues

OPLOutputBlock and XMLOutputBlock not copyable

They are stored in a packed_task and need to be copyable for that.

Test failure on 32 bit platforms

A number of the tests fail on a 32 bit platform with errors like this:

c++ -I../include -Iinclude -g -std=c++11 -Werror -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o tests test_main.o t/basic/test_changeset.cpp
--------------------------
In file included from ../include/osmium/memory/collection.hpp:40:0,
                 from ../include/osmium/osm/changeset.hpp:39,
                 from t/basic/test_changeset.cpp:3:
../include/osmium/memory/item.hpp: In function 'osmium::memory::item_size_type osmium::memory::padded_length(osmium::memory::item_size_type)':
../include/osmium/memory/item.hpp:59:31: error: redefinition of 'osmium::memory::item_size_type osmium::memory::padded_length(osmium::memory::item_size_type)'
         inline item_size_type padded_length(item_size_type length) noexcept {
                               ^
../include/osmium/memory/item.hpp:55:23: error: 'size_t osmium::memory::padded_length(size_t)' previously defined here
         inline size_t padded_length(size_t length) noexcept {
                       ^

The problem is that on a 32 bit platform size_t is uint32_t and therefore the same as item_size_type and we wind up with duplicate function definitions.

cmake build of includes

The cmake build for all includes currently builds them all in one file. The make build builds them each separately. The separate build is intentional so that we see that they work on their own, have all needed include directives etc. The cmake build should probably work the same.

Negative values from get_value_by_key

I'm using Osmium for the Open Postcode Finder and it's working great. I've recently started porting over to the new libosmium as a prototype but I'm finding a strange bug.

When running over the Geofabrik West Midlands UK PBF extract (from a few weeks ago), I come to node 1925746696. If I do something like:

QString postcode = node.tags().get_value_by_key("addr:postcode");

if(postcode.startsWith("B13 8DD")) //This should be the whole value
{
  const char* value = node.tags().get_value_by_key("addr:postcode");
  std::cout << int(value[0]) << std::endl;
  std::cout << int(value[1]) << std::endl;
  std::cout << int(value[2]) << std::endl;
  std::cout << int(value[3]) << std::endl;
  std::cout << int(value[4]) << std::endl;
  std::cout << int(value[5]) << std::endl;
  std::cout << int(value[6]) << std::endl;
  std::cout << int(value[7]) << std::endl;
  std::cout << int(value[8]) << std::endl;
  std::cout << int(value[9]) << std::endl;
  std::cout << int(value[10]) << std::endl;
}

I would expect this to output:

66
49
51
32
56
68
68
0
<something>
<something>
<something>

where <something> could be anything since it's past the end of the array (even segfault I guess). however, what is actually printed is:

66
49
51
32
56
68
68
-30
-128
-114
0

The first 7 entries match exactly "B13 8DD" but then, rather than being NULL terminated (as I would assume the char* is), it has three negative values before it has a zero.

I've found a good number of other objects in the UK with similar behaviour and it's consistently the same objects. I suppose that it's possible that it's actually what's in the OSM database but I have no way to check other that with Osmium.

I'm on openSUSE 12.3 (x86_64) using GCC 4.8.1 with the latest master of libosmium.

Thanks for the great tool.

need mkstemp replacement

The error is trivial:

..\..\test\t\index\test_typed_mmap.cpp(52): error C3861: 'mkstemp': identifier not found [D:\build_d\libosmium7\build-cmake\test\test_typed_mmap.vcxproj]

The solutions can be completely different (from avoiding mkstemp to adding its implementation from mingw into headers).

Autodetect file format

File formats (XML, PBF, ...) are currently detected based on file name suffixes. Additionally or instead we could detect formats by looking at the first few bytes of the file.

Switching to boost filesystem?

The recent changes for Windows have surfaced a few problems where code accessing the file system was Linux/OSX specific. We have fixed all those issues with the necessary #ifdefs etc. but those were all ad-hoc fixes here and there and no systematic approach. And it made the code a big uglier and harder to maintain.

There are a few ways we could take:

• Leave things as they are. After all it works and there are no extra external dependencies.
• Switch to Boost Filesystem. Makes some code cleanups possible. Introduces external dependency. But C++ standard "File System TS" will be very similar to Boost Filesystem, so change later should be very easy.
• Wait for C++ "File System TS", which is in the works and will probably released sometime this year. It might take a while to get implemented everywhere though.

@alex85k, @springmeyer, @DennisOSRM: Opinions?

Buffer::purge_removed() crashes while using MultipolygonCollector

This issue can be reproduced by setting limit in collector.hpp:499 to a low value. Then purge_removed() is called while assembling multipolygons.

stack trace:

0 osmium::memory::Item::type item.hpp 158 0x5a6bf6
1 osmium::memory::ItemIteratorosmium::OSMEntity::advance_to_next_item_of_right_type item_iterator.hpp 108 0x60a969
2 osmium::memory::ItemIteratorosmium::OSMEntity::operator++ item_iterator.hpp 130 0x624826
3 osmium::memory::Buffer::purge_removedosmium::relations::Collector<osmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>> buffer.hpp 507 0x62359d
4 osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::possibly_purge_removed_members collector.hpp 501 0x621c86
5 osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2::find_and_add_object collector.hpp 177 0x61fc6c
6 osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2::way collector.hpp 201 0x61df61
7 osmium::detail::apply_item_recurseosmium::relations::Collector<osmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2, osmium::memory::Item> visitor.hpp 72 0x61b7d3
8 osmium::detail::apply_item_recurse<m1::osm::osmium::clipping::ClipCropped::Pass2, osmium::memory::Item, osmium::relations::Collector<osmium::area::MultipolygonCollectorosmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 209 0x616f33
9 osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>, m1::osm::osmium::clipping::ClipCropped::Pass2, osmium::relations::Collectorosmium::area::MultipolygonCollector<osmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 238 0x6108f2
10 osmium::apply<osmium::io::Reader, m1::osm::osmium::clipping::ClipCropped::Pass2, osmium::relations::Collector<osmium::area::MultipolygonCollectorosmium::area::Assembler, false, true, false>::HandlerPass2> visitor.hpp 245 0x60a732
....

Uninitialized scalar field

From a recent coverity scan:

osmium/memory/item.hpp: 126 in osmium::memory::Item::Item(unsigned int, osmium::item_type)()
120             protected:
121     
122                 explicit Item(item_size_type size=0, item_type type=item_type()) noexcept :
123                     m_size(size),
124                     m_type(type),
125                     m_removed(false) {
   CID 1255394:  Uninitialized scalar field  (UNINIT_CTOR)
   Non-static class member "m_padding" is not initialized in this constructor nor in any functions that it calls.
126                 }
127     
128                 Item(const Item&) = delete;
129                 Item(Item&&) = delete;
130     
131                 Item& operator=(const Item&) = delete;

Ostream-support (operator<<)

All Osmium objects should get support for writing them to streams (ie with operator<<). (And not only ostream but basic_ostream probably.) This should be in extra header files so that those don't have to be included if you don't need them.

For many objects there are several ways they could be sent to an ostream. Maybe we only need one implementation, maybe we need some way of changing the way objects are formatted in ostreams (like with basic types such as int that can be written in decimal, hex, oct, etc.).

Add Timestamp class?

There is a helper class in utils/timestamp.hpp. But it might be better to have a real Timestamp class and use it in Object. Internally it would just contain a uint32_t, too. But this would allow us to add extra methods to it.

make doc fails

Running make doc in a clean checkout fails with:

make: *** No rule to make target `doc/xml/classosmium_1_1*', needed by `doc/classes.txt'.  Stop.

running it a second time completes successfully.

I think the problem is that the wildcard is expanded early, when the files don't yet exist. On the second run they do exist so the wildcard can match.

Windows support

There are lots of changes needed for Windows support. Smaller changes can just be added and tracked in this issue, for more complicated changes a new issue should be opened.

Changes in alex85k/libosmium/msvc:

• alex85k/libosmium@09511be - superseeded
• alex85k/libosmium@f5c60c6 - merged
• alex85k/libosmium@7ebd381 - merged
• alex85k/libosmium@7df90b7 - merged
• alex85k/libosmium@ef238bb - see #35, not merged, superseeded by 1f46a08
• alex85k/libosmium@c6eb013 - see #34, not merged, superseeded by efaffc3
• alex85k/libosmium@59763f6 - not merged, superseeded by d959a9e
• alex85k/libosmium@aba2668 - out of date
• alex85k/libosmium@e76438d - see #34, not merged, superseeded by efaffc3
• alex85k/libosmium@c951857 - merged
• alex85k/libosmium@4397732 - merged
• alex85k/libosmium@f4b0f82 - not merged, superseeded by ec037bd
• alex85k/libosmium@0cd5f27 - merged
• alex85k/libosmium@2a9f59a - merged
• alex85k/libosmium@00c2504 - not merged, superseeded by b14222c
• alex85k/libosmium@6e430a1 - superseeded by alex85k/libosmium@b397605
• alex85k/libosmium@4de4e18 - merged
• alex85k/libosmium@9a1f19d - not merged, superseeded by 28c1746
• alex85k/libosmium@2d039c3 - fixed with 32b2331
• alex85k/libosmium@c64d756 - merged
• alex85k/libosmium@b397605 - not merged, will not work like that, see #36
• alex85k/libosmium@9a34e43 - not merged, superseeded by 12bf8ca
• alex85k/libosmium@3959b20 - merged
• alex85k/libosmium@88103f5 - merged
• alex85k/libosmium@d07f43e - merged as 72844f4
• alex85k/libosmium@6c11293 - not merged, superseeded by 0f34aa3
• alex85k/libosmium@e8ad20f - not merged, superseeded by 32b2331
• alex85k/libosmium@c71b03f - merged
• alex85k/libosmium@4190c6d - merged (see #39)
• alex85k/libosmium@5931171 - merged

Other issues concerning windows are labled 'windows'.

Input/output using external programs

Currently input and output of osm.gz and osm.bz2 use the external programs gzip and bzip2. When we didn't have multithreading this wasn't such a bad idea, because at least the compressing and umcompressing was done in a separate process speeding things up. But it makes everything a little more brittle (what happens if it can't find that executable?) and, more importantly, it doesn't work on Windows, where we can't just fork the process.

Now that we have multithreading in Osmium, compressing and uncompressing should be done in extra threads using the respective libraries.

A similiar issue is the use of the "curl" executable for downloading OSM files from a URL. This should also be done in a thread using the curl or some other http library.

Handler vs visitor

There is the osmium::handler::Handler class and its derived classes and the is osmium::osm::Visitor. They do slightly different things and have different interfaces, but essentially they are the same thing. This needs to be sorted out.

avoid overhead of strftime?

strftime is accounting for many calls. This profiling output is from OS X when running the https://github.com/springmeyer/node-osmium tests:

ambiguous call when using map initializer list

I had multiple errors like this when compiling the tests

..\..\test\t\tags\test_filter.cpp(37): error C2668: 'osmium::builder::build_tag_list' : ambiguous call to overloaded function [D:\build_d\libosmium7\build-cmake\test\test_filter.vcxproj]
          D:\build_d\libosmium7\include\osmium/builder/builder_helper.hpp(89): could be 'const osmium::TagList &osmium::builder::build_tag_list(osmium::memory::Buffer &,std::function<void (osmium::builder::TagListBuilder &)>)'
          D:\build_d\libosmium7\include\osmium/builder/builder_helper.hpp(77): or       'const osmium::TagList &osmium::builder::build_tag_list(osmium::memory::Buffer &,const std::map<const char *,const char *,std::less<_Kty>,std::allocator<std::pair<const _Kty,_Ty>>> &)'
          with
          [
              _Kty=const char *
  ,            _Ty=const char *
          ]
          D:\build_d\libosmium7\include\osmium/builder/builder_helper.hpp(65): or       'const osmium::TagList &osmium::builder::build_tag_list(osmium::memory::Buffer &,std::initializer_list<std::pair<const char *,const char *>>)'
          while trying to match the argument list '(osmium::memory::Buffer, initializer-list)'

it is in

    const osmium::TagList& tag_list = osmium::builder::build_tag_list(buffer, {
        { "highway", "primary" },  // match
        { "name", "Main Street" }, // no match
        { "source", "GPS" }        // no match
    });

and I have no idea how the compiler can think that {...} is similar to std::function...

Segfault when applying NodeLocationsForWays handler

The following program creates a Segfault when applied to Geofabrik's Switzerland PBF file ( http://download.geofabrik.de/europe/switzerland-latest.osm.pbf ).

#include <osmium/index/map/dummy.hpp>
#include <osmium/index/map/sparse_table.hpp>
#include <osmium/index/map/stl_map.hpp>
#include <osmium/index/map/mmap_vector_anon.hpp>
#include <osmium/handler/node_locations_for_ways.hpp>
#include <osmium/visitor.hpp>
#include <osmium/io/any_input.hpp>
#include <osmium/handler.hpp>
#include <geos/util/IllegalArgumentException.h>
#include <geos/util/GEOSException.h>
#include <osmium/io/any_input.hpp>
#include <osmium/handler.hpp>

typedef osmium::index::map::Dummy<osmium::unsigned_object_id_type, osmium::Location> index_neg_type;
typedef osmium::index::map::SparseTable<osmium::unsigned_object_id_type, osmium::Location> index_pos_type;

typedef osmium::handler::NodeLocationsForWays<index_pos_type, index_neg_type> location_handler_type;


class MyHandler : public osmium::handler::Handler {

public:

    MyHandler() :
        osmium::handler::Handler() {
    }

}; // class MyHandler

int main(int argc, char* argv[]) {

    if (argc != 2) {
        std::cerr << "Usage: " << argv[0] << " OSMFILE\n";
        exit(1);
    }

    MyHandler handler;
    osmium::io::Reader reader(argv[1]);

    index_pos_type index_pos;
    index_neg_type index_neg;
    location_handler_type location_handler(index_pos, index_neg);

    osmium::apply(reader, location_handler, handler);

    return 0;
}

Makefile:

#CXXFLAGS += -O3
CXXFLAGS += -g
CXXFLAGS += -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

OS:=$(shell uname -s)
ifeq ($(OS),Darwin)
    CXXFLAGS += -stdlib=libc++
    LDFLAGS += -stdlib=libc++
endif

CXXFLAGS_WARNINGS := -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast

LIB_EXPAT := -lexpat
LIB_PBF   := -pthread -lz -lprotobuf-lite -losmpbf
LIB_BZIP2 := -lbz2


PROGRAMS := segfault

.PHONY: all clean

all: $(PROGRAMS)

segfault: main.cpp
    $(CXX) $(CXXFLAGS) $(CXXFLAGS_WARNINGS) -o $@ $< $(LDFLAGS) $(LIB_EXPAT) $(LIB_PBF) $(LIB_BZIP2)

clean:
    rm -f *.o core $(PROGRAMS)

gdb Output:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff675a700 (LWP 10911)]
[New Thread 0x7ffff5dd3700 (LWP 10912)]
[New Thread 0x7ffff55d2700 (LWP 10913)]
[New Thread 0x7ffff4dd1700 (LWP 10914)]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000434deb in google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >::sparsegroup (this=0x2981acb0, x=...)
    at /usr/include/google/sparsetable:795
795   sparsegroup(const sparsegroup& x) : group(0), settings(x.settings) {
(gdb) bt
#0  0x0000000000434deb in google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >::sparsegroup (this=0x2981acb0, 
    x=...) at /usr/include/google/sparsetable:795
#1  0x000000000043ac91 in google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > >::construct (this=0x7fffffffdcb0, p=0x2981acb0, val=...) at /usr/include/google/sparsehash/libc_allocator_with_realloc.h:78
#2  0x00000000004392a6 in std::allocator_traits<google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::_S_construct<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&> >(google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > >&, std::allocator_traits<google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::__construct_helper*, (google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&>&&)...) (__a=..., 
    __p=0x2981acb0) at /usr/include/c++/4.8/bits/alloc_traits.h:254
#3  0x0000000000437717 in std::allocator_traits<google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::construct<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&> >(google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > >&, google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&>*, (google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&>&&)...) (__a=..., __p=0x2981acb0) at /usr/include/c++/4.8/bits/alloc_traits.h:393
#4  0x0000000000435031 in std::__uninitialized_fill_n_a<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >*, unsigned long, google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > > (__first=0x2981acb0, __n=669, __x=..., __alloc=...)
    at /usr/include/c++/4.8/bits/stl_uninitialized.h:320
#5  0x0000000000432c1b in std::vector<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::_M_fill_insert (this=0x7fffffffdcb0, 
    __position=..., __n=669, __x=...) at /usr/include/c++/4.8/bits/vector.tcc:488
#6  0x000000000042f680 in std::vector<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::insert (this=0x7fffffffdcb0, __position=..., 
    __n=669, __x=...) at /usr/include/c++/4.8/bits/stl_vector.h:1024
#7  0x000000000042afe9 in std::vector<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::resize (this=0x7fffffffdcb0, 
    __new_size=43523432, __x=...) at /usr/include/c++/4.8/bits/stl_vector.h:687
#8  0x000000000043948c in google::sparsetable<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >::resize (this=0x7fffffffdcb0, 
    new_size=2089124703) at /usr/include/google/sparsetable:1317
#9  0x0000000000437917 in osmium::index::map::SparseTable<unsigned long, osmium::Location>::set (this=0x7fffffffdca0, id=2089114703, value=...)
    at /usr/include/osmium/index/map/sparse_table.hpp:91
#10 0x000000000043535e in osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >::node (this=0x7fffffffdc80, node=...) at /usr/include/osmium/handler/node_locations_for_ways.hpp:102
---Type <return> to continue, or q <return> to quit---
#11 0x000000000043317c in osmium::visitor::detail::switch_on_type<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, osmium::memory::Item> (visitor=..., item=...) at /usr/include/osmium/visitor.hpp:101
#12 0x000000000042f963 in osmium::visitor::detail::apply_item_recurse<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, osmium::memory::Item> (item=..., visitor=...) at /usr/include/osmium/visitor.hpp:153
#13 0x000000000042b577 in osmium::visitor::detail::apply_item_recurse<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, osmium::memory::Item, MyHandler> (item=..., visitor=...) at /usr/include/osmium/visitor.hpp:158
#14 0x0000000000425309 in osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>, osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, MyHandler> (it=..., end=...) at /usr/include/osmium/visitor.hpp:242
#15 0x000000000041ea52 in osmium::apply<osmium::io::Reader, osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, MyHandler> (source=...) at /usr/include/osmium/visitor.hpp:249
#16 0x00000000004069d6 in main (argc=2, argv=0x7fffffffdf18) at main.cpp:45
(gdb) 

List doxygen as dependency

The command:

make doc

depends on doxygen. This is not listed on the readme. Would it be a good idea to list the dependency there?

Testcases tests should use Catch

The tests in test/osm-testdata/testcases should use Catch instead of just having asserts.

area needs accessor functions for inner and outer ways

Support Mingw-w64 builds

To compile libosmium on mingw64 (on Windows or cross-platform), few changes are needed
(except building all the libraries with Mingw):
alex85k@8fb03c3

Tested on mingw-builds 4.9.1 (posix threads, seh exceptions), the full set of buld scripts (MSVC and Msys2+Mingw64) is here: https://github.com/alex85k/winbuilds

make examples fails on incompatible type error

c++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/libxml2 -o osmium_convert osmium_convert.cpp -stdlib=libc++ -lexpat -pthread -lz -lprotobuf-lite -losmpbf -lxml2
In file included from osmium_convert.cpp:14:
In file included from ../include/osmium/io/any_output.hpp:37:
../include/osmium/io/pbf_output.hpp:287:29: error: assigning to
      'Bytef *' (aka 'unsigned char *') from incompatible type
      'const unsigned char *'
  ...= reinterpret_cast<const unsigned char*>(in.c_str());
     ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [osmium_convert] Error 1

Simplify finding Osmium with dependencies (CMake)

After adding advanced FindOsmium to osmium-contrib, I noticed that it can also be included to main libosmium distribution. This will allow users to simply take FindOsmium.cmake from it sources and also make build scripts easier to maintain.

@joto Could you have a look? (and check on Linux)
alex85k@39f1d20

OSX: Make install assumes system has a root group

libosmium➟ make install
install -m 755 -g root -o root -d /usr/include
install: root: Invalid argument
make: *** [install] Error 67

Seems that this is the issue that blocks installing currently on OSX

Windows build fix: strncasecmp is a POSIX call

Windows is as always a treat. This time, the str(n)casecmp are available under a different name only. The following diff fixes compilation.

diff --git a/third_party/osmium/util/config.hpp b/third_party/osmium/util/config.hpp
index c30f02e..6c86d68 100644
--- a/third_party/osmium/util/config.hpp
+++ b/third_party/osmium/util/config.hpp
@@ -36,6 +36,11 @@ DEALINGS IN THE SOFTWARE.
 #include <cstdlib>
 #include <cstring>

+#ifdef _MSC_VER 
+#define strncasecmp _strnicmp
+#define strcasecmp _stricmp
+#endif
+
 namespace osmium {

     namespace config {

osmium_index fails with linker error on osx

c++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_index osmium_index.cpp -stdlib=libc++ -Wl,-Bstatic -lboost_program_options -Wl,-Bdynamic
ld: unknown option: -Bstatic
clang: error: linker command failed with exit code 1 (use -v to see invocation)

with c++:

examples➟ c++ -v
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix

Exception handling

The exception handling in Osmium is a bit of a mess. In some parts the usual exceptions (such as std::runtime_error) are used, some parts use special Osmium exceptions. This has to be straightened out. We have to look at all cases from the perspective of the library user and decide what the best kind of exception is in each case and what extra data is needed in the exception.

Everything that can't throw an exception should get a "noexcept".

Template-related errors

I have these errors and can not fix them myself:

D:\build_d\libosmium\include\osmium/index/map/vector.hpp(112): error C2976: 'std::vector' : too few template arguments [D:\build_d\libosmium\build-cmake\examples\osmium_serdump.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(650) : see declaration of 'std::vector'
          ..\..\examples\osmium_serdump.cpp(104) : see reference to class template instantiation 'osmium::index::map::VectorBasedSparseMap<osmium::unsigned_object_id_type,size_t,std::vector>' being compiled
D:\build_d\libosmium\include\osmium/index/map/vector.hpp(113): error C2955: 'std::vector' : use of class template requires template argument list [D:\build_d\libosmium\build-cmake\examples\osmium_serdump.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(650) : see declaration of 'std::vector'
D:\build_d\libosmium\include\osmium/index/map/vector.hpp(114): error C2955: 'std::vector' : use of class template requires template argument list [D:\build_d\libosmium\build-cmake\examples\osmium_serdump.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(650) : see declaration of 'std::vector'
D:\build_d\libosmium\include\osmium/index/map/vector.hpp(200): error C2282: 'osmium::index::map::VectorBasedSparseMap<osmium::unsigned_object_id_type,size_t,std::vector>::~VectorBasedSparseMap' cannot override 'osmium::index::map::Map<osmium::unsigned_object_id_type,size_t>::~Map' [D:\build_d\libosmium\build-cmake\examples\osmium_serdump.vcxproj]
          D:\build_d\libosmium\include\osmium/index/map.hpp(104) : 'osmium::index::map::Map<osmium::unsigned_object_id_type,size_t>::~Map' is not deleted
D:\build_d\libosmium\include\osmium/index/multimap/vector.hpp(55): error C2976: 'std::vector' : too few template arguments [D:\build_d\libosmium\build-cmake\examples\osmium_serdump.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(650) : see declaration of 'std::vector'
          ..\..\examples\osmium_serdump.cpp(110) : see reference to class template instantiation 'osmium::index::multimap::VectorBasedSparseMultimap<osmium::unsigned_object_id_type,osmium::unsigned_object_id_type,std::vector>' being compiled
D:\build_d\libosmium\include\osmium/index/multimap/vector.hpp(56): error C2955: 'std::vector' : use of class template requires template argument list [D:\build_d\libosmium\build-cmake\examples\osmium_serdump.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(650) : see declaration of 'std::vector'
D:\build_d\libosmium\include\osmium/index/multimap/vector.hpp(57): error C2955: 'std::vector' : use of class template requires template argument list [D:\build_d\libosmium\build-cmake\examples\osmium_serdump.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(650) : see declaration of 'std::vector'

My half-fixed branch (with cmake-building of examples by build.bat):
https://github.com/alex85k/libosmium

Maybe this is related:
http://stackoverflow.com/questions/24017466/is-it-required-to-explicitly-list-default-parameters-when-using-template-templat

OSMFile, Meta

There is a bit of overlap between what OSMFile and Meta classes are doing. And it is unclear what the relationship between OSMFile and the input and output (or Reader and Writer classes, respectively) is. The Reader class can already be instantiated with just a filename.

The meta class also might need some more data in it. It is already a bit of a hodge-podge class, but we have to put the data from the PBF header block and XML file somewhere.

This all needs to be straightened out.

Struct packing with MSVC

Struct packing in MSVC is different than with GCC and clang. (Also see #30.)

@alex85k, @DennisOSRM: Can you try changing https://github.com/osmcode/libosmium/blob/master/include/osmium/memory/item.hpp#L104 ff. to:

            item_type m_removed : 1;
            item_type m_padding : 15;

and see whether the static_assert on https://github.com/osmcode/libosmium/blob/master/include/osmium/memory/item.hpp#L171 is okay then?

(Good explanation is here: http://randomascii.wordpress.com/2010/06/06/bit-field-packing-with-visual-c/)

Test failure reading PBF and XML input

The reader test is failing for me seemingly for both PBF and XML input, with gcc 4.8.3 and the git master version of osmpbf:

Checking \033[1mtest/t/io/test_reader.cpp\033[0m ..................... \033[31m[COMPILE ERROR]\033[0m
==========================
c++ -I../include -Iinclude -g -std=c++11 -Werror -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o tests test_main.o t/io/test_reader.cpp -lbz2 -lexpat -lz -pthread -lz -lprotobuf-lite -losmpbf
--------------------------
In file included from ../include/osmium/io/detail/input_format.hpp:43:0,
                 from ../include/osmium/io/reader.hpp:55,
                 from ../include/osmium/io/pbf_input.hpp:36,
                 from ../include/osmium/io/any_input.hpp:38,
                 from t/io/test_reader.cpp:4:
../include/osmium/io/file.hpp:150:13: error: unused parameter ‘other’ [-Werror=unused-parameter]
             File(const File& other) = default;
             ^
In file included from ../include/osmium/io/reader.hpp:55:0,
                 from ../include/osmium/io/pbf_input.hpp:36,
                 from ../include/osmium/io/any_input.hpp:38,
                 from t/io/test_reader.cpp:4:
../include/osmium/io/detail/input_format.hpp: In constructor ‘osmium::io::detail::InputFormat::InputFormat(const osmium::io::File&, osmium::osm_entity_bits::type, osmium::thread::Queue<std::basic_string<char> >&)’:
../include/osmium/io/detail/input_format.hpp:78:46: note: synthesized method ‘osmium::io::File::File(const osmium::io::File&)’ first required here 
                     m_input_queue(input_queue) {
                                              ^
In file included from ../include/osmium/io/xml_input.hpp:37:0,
                 from ../include/osmium/io/any_input.hpp:39,
                 from t/io/test_reader.cpp:4:
../include/osmium/io/detail/xml_input_format.hpp: In instantiation of ‘std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_Impl::_Impl(_Fn2&&, const _Alloc&) [with _Fn2 = osmium::io::detail::XMLParser; _Fn = osmium::io::detail::XMLParser; _Alloc = std::allocator<int>; _Res = bool; _Args = {}]’:
/usr/include/c++/4.8.3/future:1275:41:   required from ‘std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_Task_state(_Fn2&&, const _Alloc&) [with _Fn2 = osmium::io::detail::XMLParser; _Fn = osmium::io::detail::XMLParser; _Alloc = std::allocator<int>; _Res = bool; _Args = {}]’
/usr/include/c++/4.8.3/ext/new_allocator.h:120:4:   required from ‘void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = std::__future_base::_Task_state<osmium::io::detail::XMLParser, std::allocator<int>, bool()>; _Args = {osmium::io::detail::XMLParser, const std::allocator<int>&}; _Tp = int]’
/usr/include/c++/4.8.3/bits/alloc_traits.h:254:4:   required from ‘static typename std::enable_if<std::allocator_traits<_Alloc>::__construct_helper<_Tp, _Args>::value, void>::type std::allocator_traits<_Alloc>::_S_construct(_Alloc&, _Tp*, _Args&& ...) [with _Tp = std::__future_base::_Task_state<osmium::io::detail::XMLParser, std::allocator<int>, bool()>; _Args = {osmium::io::detail::XMLParser, const std::allocator<int>&}; _Alloc = std::allocator<int>; typename std::enable_if<std::allocator_traits<_Alloc>::__construct_helper<_Tp, _Args>::value, void>::type = void]’
/usr/include/c++/4.8.3/bits/alloc_traits.h:393:57:   required from ‘static decltype (_S_construct(__a, __p, (forward<_Args>)(std::allocator_traits::construct::__args)...)) std::allocator_traits<_Alloc>::construct(_Alloc&, _Tp*, _Args&& ...) [with _Tp = std::__future_base::_Task_state<osmium::io::detail::XMLParser, std::allocator<int>, bool()>; _Args = {osmium::io::detail::XMLParser, const std::allocator<int>&}; _Alloc = std::allocator<int>; decltype (_S_construct(__a, __p, (forward<_Args>)(std::allocator_traits::construct::__args)...)) = <type error>]’
/usr/include/c++/4.8.3/bits/shared_ptr_base.h:400:38:   required from ‘std::_Sp_counted_ptr_inplace<_Tp, _Alloc, _Lp>::_Sp_counted_ptr_inplace(_Alloc, _Args&& ...) [with _Args = {osmium::io::detail::XMLParser, const std::allocator<int>&}; _Tp = std::__future_base::_Task_state<osmium::io::detail::XMLParser, std::allocator<int>, bool()>; _Alloc = std::allocator<int>; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2u]’
/usr/include/c++/4.8.3/ext/new_allocator.h:120:4:   [ skipping 6 instantiation contexts, use -ftemplate-backtrace-limit=0 to disable ]
/usr/include/c++/4.8.3/bits/shared_ptr.h:598:39:   required from ‘std::shared_ptr<_Tp1> std::allocate_shared(const _Alloc&, _Args&& ...) [with _Tp = std::__future_base::_Task_state<osmium::io::detail::XMLParser, std::allocator<int>, bool()>; _Alloc = std::allocator<int>; _Args = {osmium::io::detail::XMLParser, const std::allocator<int>&}]’
/usr/include/c++/4.8.3/future:1318:76:   required from ‘std::shared_ptr<std::__future_base::_Task_state_base<_Signature> > std::__create_task_state(_Fn&&, const _Alloc&) [with _Signature = bool(); _Fn = osmium::io::detail::XMLParser; _Alloc = std::allocator<int>]’
/usr/include/c++/4.8.3/future:1370:35:   required from ‘std::packaged_task<_Res(_ArgTypes ...)>::packaged_task(std::allocator_arg_t, const _Alloc&, _Fn&&) [with _Fn = osmium::io::detail::XMLParser; _Alloc = std::allocator<int>; <template-parameter-2-3> = void; _Res = bool; _ArgTypes = {}]’
/usr/include/c++/4.8.3/future:1360:27:   required from ‘std::packaged_task<_Res(_ArgTypes ...)>::packaged_task(_Fn&&) [with _Fn = osmium::io::detail::XMLParser; <template-parameter-2-2> = void; _Res = bool; _ArgTypes = {}]’
../include/osmium/thread/checked_task.hpp:55:85:   required from ‘osmium::thread::CheckedTask<T>::CheckedTask(TArgs&& ...) [with TArgs = {osmium::thread::Queue<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, osmium::thread::Queue<osmium::memory::Buffer>&, std::promise<osmium::io::Header>&, osmium::osm_entity_bits::type&, std::atomic<bool>&}; T = osmium::io::detail::XMLParser]’
../include/osmium/io/detail/xml_input_format.hpp:686:102:   required from here
../include/osmium/io/detail/xml_input_format.hpp:305:17: error: unused parameter ‘other’ [-Werror=unused-parameter]
                 XMLParser(XMLParser&& other) = default;
                 ^
In file included from ../include/osmium/thread/checked_task.hpp:37:0,
                 from ../include/osmium/io/reader.hpp:62,
                 from ../include/osmium/io/pbf_input.hpp:36,
                 from ../include/osmium/io/any_input.hpp:38,
                 from t/io/test_reader.cpp:4:
/usr/include/c++/4.8.3/future:1307:49: note: synthesized method ‘osmium::io::detail::XMLParser::XMLParser(osmium::io::detail::XMLParser&&)’ first required here 
    : _Alloc(__a), _M_fn(std::forward<_Fn2>(__fn)) { }
                                                 ^
cc1plus: all warnings being treated as errors
==========================

OS X build issues

tests

1. endian.h not available on OS X

In file included from t/geom/test_geos_wkb.cpp:5:
../include/osmium/geom/wkb.hpp:44:11: fatal error: 'endian.h' file not found
# include <endian.h>

2. -Werror,-Wsign-compare

In file included from t/basic/test_node.cpp:1:
include/catch.hpp:831:34: error: comparison of integers of different signs: 'int' and 'unsigned long long' [-Werror,-Wsign-compare]
            return opCast( lhs ) ==  opCast( rhs );
                   ~~~~~~~~~~~~~ ^   ~~~~~~~~~~~~~
include/catch.hpp:876:39: note: in instantiation of member function 'Catch::Internal::Evaluator<int, unsigned long long, 0>::evaluate' requested here
        return Evaluator<T1, T2, Op>::evaluate( lhs, rhs );
                                      ^
include/catch.hpp:1293:39: note: in instantiation of function template specialization 'Catch::Internal::compare<0, int, unsigned long long>' requested here
            .setResultType( Internal::compare<Op>( m_lhs, rhs ) )
                                      ^
include/catch.hpp:1236:16: note: in instantiation of function template specialization 'Catch::ExpressionLhs<const int &>::captureExpression<0, unsigned long long>' requested here
        return captureExpression<Internal::IsEqualTo>( rhs );
               ^
t/basic/test_node.cpp:47:15: note: in instantiation of function template specialization 'Catch::ExpressionLhs<const int &>::operator==<unsigned long long>' requested here
    REQUIRE(0 == node.positive_id());
              ^

examples

1. -Werror,-Wunused-variable

In file included from osmium_convert.cpp:14:
In file included from ../include/osmium/io/any_output.hpp:38:
In file included from ../include/osmium/io/opl_output.hpp:37:
../include/osmium/io/detail/opl_output_format.hpp:109:25: error: unused variable 'len' [-Werror,-Wunused-variable]
                    int len = snprintf(m_tmp_buffer, tmp_buffer_size, format, std::forward<TArgs>(args)...);
                        ^
In file included from osmium_convert.cpp:14:
In file included from ../include/osmium/io/any_output.hpp:40:
In file included from ../include/osmium/io/xml_output.hpp:37:
../include/osmium/io/detail/xml_output_format.hpp:100:25: error: unused variable 'len' [-Werror,-Wunused-variable]
                    int len = snprintf(buffer, max_size, format, value);
                        ^
2 errors generated.
make: *** [osmium_convert] Error 1

Make mmap_vector_* classes implement fill STL vector interface

The classes osmium::detail::mmap_vector_base/anon/file currently only implement that part of the STL vector interface that we need. It would be nice to implement that interface fully and add tests for it.

Add multipolygon support

The new Osmium doesn't support multipolygons yet.

How I managed to make libosmium work on my MacBook

I am on OS X 10.9.2, aka as Mavericks and I am using Macports

a) What I installed following the list of prerequisites (package @Version):
boost @1.55.0 # there are no other boost packages, no boost-program-options, no libboost-test
zlib @1.2.8
bzip2 @1.0.6
gdal # There is a Macport package of gdal, but as a serious GIS worker, I have gdal from trunk, of course ;-)
expat @2.1.0
geos @3.4.2
sparsehash @2.0.2
protobuf-cpp @2.5.0
Doxygen # who wants to read documentation ?
libboost-test # see boost
OSMPBF from github

b) Macports install into /opt/local, so I had to point to /opt/local/lib and /opt/local/include in various places

c) For some reason which I read months ago and now forgot: one has to add the suffix "-mt" in 2 places:
examples/Makefile: LIB_PRGOPT := -lboost_program_options-mt
test/run_tests.sh: PARAMS= ... -lboost_unit_test_framework-mt"

d) To make it really work I also have to remove a suffix: -lprotobuf instead of -lprotobuf-lite
examples/Makefile: LIB_PBF := -pthread -lz -lprotobuf -losmpbf

Outcome #1:

[libosmium]> make test
(cd test && ./run_tests.sh)
Checking t/area/test_node_ref_segment.cpp ...
c++ -I../include -I. -g -std=c++11 -stdlib=libc++ -Wall -Wextra -Wredundant-decls -Wdisabled-optimization -pedantic -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -o tests -L/Users/peifer/local/lib -L/opt/local/lib test_main.cpp test_utils.cpp t/area/test_node_ref_segment.cpp -DBOOST_TEST_DYN_LINK -lboost_unit_test_framework-mt
Running 5 test cases...
No errors detected

Then more tests and more of "No errors detected"

Outcome #2: some warnings which don't seem to harm

[examples]> CPLUS_INCLUDE_PATH=/opt/local/include make
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_convert osmium_convert.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lexpat -pthread -lz -lprotobuf -losmpbf -lz -lbz2
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_debug osmium_debug.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lexpat -pthread -lz -lprotobuf -losmpbf -lz -lbz2
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_index osmium_index.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lboost_program_options-mt
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_mpdump osmium_mpdump.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lexpat -pthread -lz -lprotobuf -losmpbf -lz -lbz2
In file included from osmium_mpdump.cpp:17:
In file included from ../include/osmium/area/collector.hpp:44:
../include/osmium/relations/collector.hpp:163:57: warning: comparison of integers of different signs: 'int' and 'size_t' (aka 'unsigned long')
[-Wsign-compare]
assert(member_meta.member_pos() < m_collector.get_relation(relation_meta).members().size());
~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/assert.h:93:25: note: expanded from macro 'assert'
(builtin_expect(!(e), 0) ? __assert_rtn(__func, FILE, LINE, #e) : (void)0)
^
../include/osmium/relations/collector.hpp:198:31: note: in instantiation of member function
'osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2::find_and_add_object'
requested here
if (! find_and_add_object(node)) {
^
../include/osmium/visitor.hpp:111:33: note: in instantiation of member function
'osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2::node' requested here
visitor.node(static_cast<MaybeConst<TItem, osmium::Node>&>(item));
^
../include/osmium/visitor.hpp:172:17: note: in instantiation of function template specialization
'osmium::visitor::detail::switch_on_typeosmium::relations::Collector<osmium::area::Collector<osmium::area::Assembler, false, true,
false>::HandlerPass2, osmium::memory::Item>' requested here
switch_on_type(visitor, item, std::is_base_of<osmium::handler::Handler, TVisitor>());
^
../include/osmium/visitor.hpp:178:17: note: in instantiation of function template specialization
'osmium::visitor::detail::apply_item_recurseosmium::relations::Collector<osmium::area::Collector<osmium::area::Assembler, false, true,
false>::HandlerPass2, osmium::memory::Item>' requested here
apply_item_recurse(item, more...);
^
../include/osmium/visitor.hpp:261:38: note: in instantiation of function template specialization
'osmium::visitor::detail::apply_item_recurse<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long long,
osmium::Location>, osmium::index::map::Dummy<unsigned long long, osmium::Location> >, osmium::memory::Item,
osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2>' requested here
osmium::visitor::detail::apply_item_recurse(_it, visitors...);
^
../include/osmium/visitor.hpp:268:9: note: in instantiation of function template specialization
'osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>,
osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long long, osmium::Location>,
osmium::index::map::Dummy<unsigned long long, osmium::Location> >,
osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2>' requested here
apply(osmium::io::InputIterator {source},
^
osmium_mpdump.cpp:70:13: note: in instantiation of function template specialization 'osmium::apply<osmium::io::Reader,
osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long long, osmium::Location>,
osmium::index::map::Dummy<unsigned long long, osmium::Location> >,
osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2>' requested here
osmium::apply(reader2, location_handler, collector.handler());
^
1 warning generated.
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_read osmium_read.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lexpat -pthread -lz -lprotobuf -losmpbf -lz -lbz2
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_serdump osmium_serdump.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lexpat -pthread -lz -lprotobuf -losmpbf -lz -lbz2
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -I/Users/peifer/local/include -o osmium_toogr osmium_toogr.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lexpat -pthread -lz -lprotobuf -losmpbf -lz -lbz2 -L/Users/peifer/local/lib -lgdal
clang++ -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -stdlib=libc++ -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -I/Users/peifer/local/include -o osmium_toogr2 osmium_toogr2.cpp -L/opt/local/lib -L/Users/peifer/local/lib -stdlib=libc++ -lexpat -pthread -lz -lprotobuf -losmpbf -lz -lbz2 -L/Users/peifer/local/lib -lgdal
In file included from osmium_toogr2.cpp:31:
In file included from ../include/osmium/area/collector.hpp:44:
../include/osmium/relations/collector.hpp:163:57: warning: comparison of integers of different signs: 'int' and 'size_t' (aka 'unsigned long')
[-Wsign-compare]
assert(member_meta.member_pos() < m_collector.get_relation(relation_meta).members().size());
~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/assert.h:93:25: note: expanded from macro 'assert'
(builtin_expect(!(e), 0) ? __assert_rtn(__func, FILE, LINE, #e) : (void)0)
^
../include/osmium/relations/collector.hpp:198:31: note: in instantiation of member function
'osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2::find_and_add_object'
requested here
if (! find_and_add_object(node)) {
^
../include/osmium/visitor.hpp:111:33: note: in instantiation of member function
'osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2::node' requested here
visitor.node(static_cast<MaybeConst<TItem, osmium::Node>&>(item));
^
../include/osmium/visitor.hpp:172:17: note: in instantiation of function template specialization
'osmium::visitor::detail::switch_on_typeosmium::relations::Collector<osmium::area::Collector<osmium::area::Assembler, false, true,
false>::HandlerPass2, osmium::memory::Item>' requested here
switch_on_type(visitor, item, std::is_base_of<osmium::handler::Handler, TVisitor>());
^
../include/osmium/visitor.hpp:178:17: note: in instantiation of function template specialization
'osmium::visitor::detail::apply_item_recurseosmium::relations::Collector<osmium::area::Collector<osmium::area::Assembler, false, true,
false>::HandlerPass2, osmium::memory::Item>' requested here
apply_item_recurse(item, more...);
^
../include/osmium/visitor.hpp:178:17: note: in instantiation of function template specialization
'osmium::visitor::detail::apply_item_recurse<MyOGRHandler, osmium::memory::Item,
osmium::relations::Collector<osmium::area::Collectorosmium::area::Assembler, false, true, false>::HandlerPass2>' requested here
apply_item_recurse(item, more...);
^
../include/osmium/visitor.hpp:261:38: note: in instantiation of function template specialization
'osmium::visitor::detail::apply_item_recurse<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long long,
osmium::Location>, osmium::index::map::Dummy<unsigned long long, osmium::Location> >, osmium::memory::Item, MyOGRHandler,
osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2>' requested here
osmium::visitor::detail::apply_item_recurse(_it, visitors...);
^
../include/osmium/visitor.hpp:268:9: note: in instantiation of function template specialization
'osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>,
osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long long, osmium::Location>,
osmium::index::map::Dummy<unsigned long long, osmium::Location> >, MyOGRHandler,
osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2>' requested here
apply(osmium::io::InputIterator {source},
^
osmium_toogr2.cpp:315:13: note: in instantiation of function template specialization 'osmium::apply<osmium::io::Reader,
osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long long, osmium::Location>,
osmium::index::map::Dummy<unsigned long long, osmium::Location> >, MyOGRHandler,
osmium::relations::Collectorosmium::area::Collector<osmium::area::Assembler, false, true, false>::HandlerPass2>' requested here
osmium::apply(reader2, location_handler, ogr_handler, collector.handler());
^
1 warning generated.

Projects created with make_osmium_project.sh don't compile

If one uses the wizard-like make_osmium_project.sh, the resulting code doesn't compile (anymore).

Add multithreading to OSM output

The output of OSM data into files should use multithreading.

pbf.hpp:62:5: warning: control reaches end of non-void function

/home/dennis/Coding/Project-OSRM/ThirdParty/osmium/io/file_compression.hpp: In function ‘const char* osmium::io::as_string(osmium::io::file_compression)’:
/home/dennis/Coding/Project-OSRM/ThirdParty/osmium/io/file_compression.hpp:57:9: warning: control reaches end of non-void function [-Wreturn-type]

binary '=' - no operator found

Several errors occur while compiling tests:

C:\Program Files (x86)\Microsoft Visual C++ Compiler Nov 2013 CTP\include\utility(53): error C2678: binary '=' : no operator found which takes a left-hand operand of type 'const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>' (or there is no acceptable conversion) [D:\build_d\libosmium7\build-cmake\test\test_geojson.vcxproj]
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(948): could be 'std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>> &std::vector<_Ty,std::allocator<_Ty>>::operator =(const std::vector<_Ty,std::allocator<_Ty>> &)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(926): or       'std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>> &std::vector<_Ty,std::allocator<_Ty>>::operator =(std::initializer_list<std::pair<osmium::object_id_type,osmium::Location>>)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(822): or       'std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>> &std::vector<_Ty,std::allocator<_Ty>>::operator =(std::vector<_Ty,std::allocator<_Ty>> &&)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          while trying to match the argument list '(const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>, const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual C++ Compiler Nov 2013 CTP\include\utility(61) : see reference to function template instantiation 'void std::swap<_Ty>(_Ty &,_Ty &)' being compiled
          with
          [
              _Ty=const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<std::pair<osmium::object_id_type,osmium::Location>>>
          ]
          C:\Program Files (x86)\Microsoft Visual C++ Compiler Nov 2013 CTP\include\utility(189) : see reference to function template instantiation 'void std::_Swap_adl<_Ty2>(_Ty &,_Ty &)' being compiled
          with
          [
              _Ty2=const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<std::pair<osmium::object_id_type,osmium::Location>>>
  ,            _Ty=const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<std::pair<osmium::object_id_type,osmium::Location>>>
          ]
          C:\Program Files (x86)\Microsoft Visual C++ Compiler Nov 2013 CTP\include\utility(185) : while compiling class template member function 'void std::pair<bool,const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>>::swap(std::pair<bool,const std::vector<_Ty,std::allocator<_Ty>>> &)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual C++ Compiler Nov 2013 CTP\include\utility(211) : see reference to function template instantiation 'void std::pair<bool,const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>>::swap(std::pair<bool,const std::vector<_Ty,std::allocator<_Ty>>> &)' being compiled
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          d:\build_d\libosmium7\test\t\geom\../basic/helper.hpp(73) : see reference to class template instantiation 'std::pair<bool,const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>>' being compiled
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
C:\Program Files (x86)\Microsoft Visual C++ Compiler Nov 2013 CTP\include\utility(54): error C2678: binary '=' : no operator found which takes a left-hand operand of type 'const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>' (or there is no acceptable conversion) [D:\build_d\libosmium7\build-cmake\test\test_geojson.vcxproj]
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(948): could be 'std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>> &std::vector<_Ty,std::allocator<_Ty>>::operator =(const std::vector<_Ty,std::allocator<_Ty>> &)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(926): or       'std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>> &std::vector<_Ty,std::allocator<_Ty>>::operator =(std::initializer_list<std::pair<osmium::object_id_type,osmium::Location>>)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\vector(822): or       'std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>> &std::vector<_Ty,std::allocator<_Ty>>::operator =(std::vector<_Ty,std::allocator<_Ty>> &&)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
          while trying to match the argument list '(const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>, const std::vector<std::pair<osmium::object_id_type,osmium::Location>,std::allocator<_Ty>>)'
          with
          [
              _Ty=std::pair<osmium::object_id_type,osmium::Location>
          ]
  test_geos.cpp

Some vector-pair-template-powered assignment that is melting my old-style Java-C brain.std::pair

static_assert(sizeof(Item) == 8, "Class osmium::Item has wrong size!");

The struct packing on Windows is different to what GCC/Clang do. Checking struct sizes will thus fail.

Happening here: https://github.com/osmcode/libosmium/blob/master/include/osmium/memory/item.hpp#L171

On Windows only fields of the same underlying type will be packed. While this is inconvenient, the standard says packing is implementation defined.

Compile error with g++-4.8

Seeing that clang++ is working with master branch but not g++, on travis (https://travis-ci.org/osmcode/libosmium/jobs/18206306#L581).

g++-4.8 -O3 -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I../include -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast -o osmium_mpdump osmium_mpdump.cpp  -lexpat -pthread -lz -lprotobuf-lite -losmpbf -lz -lbz2
In file included from ../include/osmium/relations/collector.hpp:51:0,
                 from ../include/osmium/area/collector.hpp:44,
                 from osmium_mpdump.cpp:17:
../include/osmium/relations/detail/member_meta.hpp: In function ‘constexpr bool osmium::relations::operator<(const osmium::relations::MemberMeta&, const osmium::relations::MemberMeta&)’:
../include/osmium/relations/detail/member_meta.hpp:102:32: error: call to non-constexpr function ‘osmium::object_id_type osmium::relations::MemberMeta::member_id() const’

cmake test build doesn't build stuff in osm-testdata

Unlike the make build, the cmake build doesn't build/run the tests in test/osm-testdata.

Templates vs. virtual derived classes in index::map/multimap classes

The classes in osmium::index::map/multimap are just simple wrapper classes for different data structures. Most of their functions are really simple and good candidates for inlining. This would work if thoses classes are simple template classes. But in some cases we also want to use them polymorphically, ie we want to decide at runtime which implementation to use depending on the kind of OSM data we work with (mainly whole planet vs. small extract) and how much memory we have available. For that they have to be virtual derived classes.

This becomes worse for the multimap classes, because their get_all() functions returns two iterators through which we can access the results. But the type of those iterators is different depending on the implementation of the multimap class. To make this work polymorphically, the iterators would have to be polymorphic, too. Of course all of this slows down the use case where you already know at compile time what class you want.

The currently situation is a strange mix. The multimap classes are virtual derived classes, but some of their functions (for instance get_all()) are not virtual and they have different return types in the different derived classes. This is not only ugly, it is error prone and hard to work with and will not work polymorphically.

One option would be to split the functionality into one class hierarchy that can not be used polymorphically (so you have to decide at compile time) and another class hierarchy on top of that that is usable polymorphically.

And maybe all these complications are not really problematic, because the cost of not inlining but having those virtual function calls isn't all that great compared to what else Osmium is doing. Maybe we should benchmark that... Unfortunately we have to implement both variants to do that...

To decide on the right design, we might have to look at more use cases.

MSVC constexpr issues

It seems MSVC does not support constexpr properly. Instead of littering the code with ifdefs, does it make sense to define a 'OSMIUM_CONSTEXPR' once depending on compiler and use that? Are all cases of constexpr problematic, or only some of them?

@alex85k: Opinions?

On Windows multipolygon tests don't compare output with reference.

This comparison uses a ruby program, a shell script and sqlite. We'll need some other way of doing this.

packaged_task<void()> errors

I have these errors and can not fix them myself:

C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1014): error C2182: '_Get_value' : illegal use of type 'void' [D:\build_d\libosmium\build-cmake\examples\osmium_area_test.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1466) : see reference to class template instantiation 'std::_State_manager<_Ty>' being compiled
          with
          [
              _Ty=void
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1747) : see reference to class template instantiation 'std::_Promise<_Ret>' being compiled
          with
          [
              _Ret=void
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1747) : while compiling class template member function 'std::packaged_task<void (void)>::packaged_task(std::packaged_task<void (void)> &&) throw()'
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\type_traits(1545) : see reference to function template instantiation 'std::packaged_task<void (void)>::packaged_task(std::packaged_task<void (void)> &&) throw()' being compiled
          D:\build_d\libosmium\include\osmium/thread/checked_task.hpp(55) : see reference to class template instantiation 'std::packaged_task<void (void)>' being compiled
          D:\build_d\libosmium\include\osmium/io/reader.hpp(176) : see reference to function template instantiation 'osmium::thread::CheckedTask<osmium::io::detail::ReadThread>::CheckedTask<osmium::thread::Queue<std::string>&,osmium::io::Decompressor*,std::atomic<bool>&>(osmium::thread::Queue<std::string> &,osmium::io::Decompressor *&&,std::atomic<bool> &)' being compiled
          D:\build_d\libosmium\include\osmium/io/reader.hpp(176) : see reference to function template instantiation 'osmium::thread::CheckedTask<osmium::io::detail::ReadThread>::CheckedTask<osmium::thread::Queue<std::string>&,osmium::io::Decompressor*,std::atomic<bool>&>(osmium::thread::Queue<std::string> &,osmium::io::Decompressor *&&,std::atomic<bool> &)' being compiled
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1021): error C2182: '_Val' : illegal use of type 'void' [D:\build_d\libosmium\build-cmake\examples\osmium_area_test.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1029): error C2182: '_Val' : illegal use of type 'void' [D:\build_d\libosmium\build-cmake\examples\osmium_area_test.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1747): error C2665: 'std::forward' : none of the 2 overloads could convert all the argument types [D:\build_d\libosmium\build-cmake\examples\osmium_area_test.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\type_traits(1509): could be '_Ty &&std::forward<std::_Promise<_Ret>>(std::_Promise<_Ret> &&) throw()'
          with
          [
              _Ty=std::_Promise<void>
  ,            _Ret=void
          ]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\type_traits(1503): or       '_Ty &&std::forward<std::_Promise<_Ret>>(std::_Promise<_Ret> &)'
          with
          [
              _Ty=std::_Promise<void>
  ,            _Ret=void
          ]
          while trying to match the argument list '(std::_Promise<int>)'
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\future(1749): error C2512: 'std::_Promise<int>' : no appropriate default constructor available [D:\build_d\libosmium\build-cmake\examples\osmium_area_test.vcxproj]
  osmium_convert.cpp

This is known to be an error or standard unclearness about packaged_task<void()>:
http://stackoverflow.com/questions/14744588/why-is-stdpackaged-taskvoid-not-valid

We had to add a workaround for this in OSRM:
Project-OSRM/osrm-backend@42d3ee9
Project-OSRM/osrm-backend#998 (comment)
in libosmium it seems much harder...

My half-fixed branch (with cmake-building of examples by build.bat):
https://github.com/alex85k/libosmium

Compile error: debian based distro

Hey @joto,

I am seeing this issue on Travis (link to log):

In file included from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/pbf_input.hpp:37:0,
                 from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_input.hpp:38,
                 from /home/travis/build/Project-OSRM/osrm-backend/Extractor/Extractor.cpp:46:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp: In member function ‘void osmium::io::detail::HeaderBlobParser::handle_blob(const string&)’:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp:480:42: error: ‘class OSMPBF::HeaderBlock’ has no member named ‘has_osmosis_replication_timestamp’
                     if (pbf_header_block.has_osmosis_replication_timestamp()) {
                                          ^
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp:481:106: error: ‘class OSMPBF::HeaderBlock’ has no member named ‘osmosis_replication_timestamp’
                         m_header.set("osmosis_replication_timestamp", osmium::Timestamp(pbf_header_block.osmosis_replication_timestamp()).to_iso());
                                                                                                          ^
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp:484:42: error: ‘class OSMPBF::HeaderBlock’ has no member named ‘has_osmosis_replication_sequence_number’
                     if (pbf_header_block.has_osmosis_replication_sequence_number()) {
                                          ^
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp:485:109: error: ‘class OSMPBF::HeaderBlock’ has no member named ‘osmosis_replication_sequence_number’
                         m_header.set("osmosis_replication_sequence_number", std::to_string(pbf_header_block.osmosis_replication_sequence_number()));
                                                                                                             ^
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp:488:42: error: ‘class OSMPBF::HeaderBlock’ has no member named ‘has_osmosis_replication_base_url’
                     if (pbf_header_block.has_osmosis_replication_base_url()) {
                                          ^
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp:489:87: error: ‘class OSMPBF::HeaderBlock’ has no member named ‘osmosis_replication_base_url’
                         m_header.set("osmosis_replication_base_url", pbf_header_block.osmosis_replication_base_url());
                                                                                       ^
In file included from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/bzip2_compression.hpp:45:0,
                 from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_compression.hpp:36,
                 from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_input.hpp:36,
                 from /home/travis/build/Project-OSRM/osrm-backend/Extractor/Extractor.cpp:46:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/compression.hpp: At global scope:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/compression.hpp:224:24: warning: ‘osmium::io::{anonymous}::registered_no_compression’ defined but not used [-Wunused-variable]
             const bool registered_no_compression = osmium::io::CompressionFactory::instance().register_compression(osmium::io::file_compression::none,
                        ^
In file included from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_compression.hpp:36:0,
                 from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_input.hpp:36,
                 from /home/travis/build/Project-OSRM/osrm-backend/Extractor/Extractor.cpp:46:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/bzip2_compression.hpp:182:24: warning: ‘osmium::io::{anonymous}::registered_bzip2_compression’ defined but not used [-Wunused-variable]
             const bool registered_bzip2_compression = osmium::io::CompressionFactory::instance().register_compression(osmium::io::file_compression::bzip2,
                        ^
In file included from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_compression.hpp:37:0,
                 from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_input.hpp:36,
                 from /home/travis/build/Project-OSRM/osrm-backend/Extractor/Extractor.cpp:46:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/gzip_compression.hpp:121:24: warning: ‘osmium::io::{anonymous}::registered_gzip_compression’ defined but not used [-Wunused-variable]
             const bool registered_gzip_compression = osmium::io::CompressionFactory::instance().register_compression(osmium::io::file_compression::gzip,
                        ^
In file included from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/pbf_input.hpp:37:0,
                 from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_input.hpp:38,
                 from /home/travis/build/Project-OSRM/osrm-backend/Extractor/Extractor.cpp:46:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/pbf_input_format.hpp:664:28: warning: ‘osmium::io::detail::{anonymous}::registered_pbf_input’ defined but not used [-Wunused-variable]
                 const bool registered_pbf_input = osmium::io::detail::InputFormatFactory::instance().register_input_format(osmium::io::file_format::pbf,
                            ^
In file included from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/xml_input.hpp:37:0,
                 from /home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/any_input.hpp:39,
                 from /home/travis/build/Project-OSRM/osrm-backend/Extractor/Extractor.cpp:46:
/home/travis/build/Project-OSRM/osrm-backend/ThirdParty/osmium/io/detail/xml_input_format.hpp:627:28: warning: ‘osmium::io::detail::{anonymous}::registered_xml_input’ defined but not used [-Wunused-variable]
                 const bool registered_xml_input = osmium::io::detail::InputFormatFactory::instance().register_input_format(osmium::io::file_format::xml,
                            ^

Looks like the PBF definitions are too old?

Appveyor config

I have added an appveyor config, but it is only a start and it doesn't work yet. See appveyor. I don't understand enough about windows builds and how to use the nuget stuff so that cmake will find the libraries. The config is just bits and pieces stolen from other configs so maybe I am doing it all wrong.

@alex85k: Maybe you can take a look.

profiling of pbf parsing

Ticket to track ideas around optimizing threaded pbf parsing.

Unclear how much this impacts linux, but at least on OS X we are seeing std::vector<unsigned char> allocation taking a surprising amount of time: