mapbox / vtzero

Minimalist vector tile decoder and encoder in C++

vtzero

Tiny and fast vector tile decoder and encoder in C++.

Implements the Mapbox Vector Tile Specification 2.x.

Depends

• C++11 compiler (GCC 4.8 or higher, clang 3.5 or higher, ...)
• CMake
• Protozero version >= 1.7.0

Build

First clone protozero:

git clone [email protected]:mapbox/protozero.git

Then clone vtzero beside protozero. The vtzero build system will, among several places, look for protozero at ../protozero. (If you would like to use protozero from a different path you can set PROTOZERO_INCLUDE_DIR in the CMake configuration step.)

Then, inside the vtzero directory do:

git submodule update --init

Finally, to build the examples and tests do:

mkdir build
cd build
cmake ..
make

Call ctest to run the tests.

Examples

Several examples are provided to show usage of the library.

Call

examples/vtzero-create

to create test tile named test.mvt.

Call

examples/vtzero-show TILE-FILE

to show the contents of TILE-FILE.

You can use

examples/vtzero-check TILE-FILE

to check vector tile for validity.

Docs

Extensive documentation is available:

• Tutorial (start here)
• Reading vector tiles
• Writing vector tiles
• Advanced vtzero topics

Make sure to read all of it before using vtzero. Vtzero isn't the simplest library to use, because it always chooses performance over ease of use.

If Doxygen is installed on your system, the build process will automatically create the API docs for you. The results will be in your build directory under doc/html.

Authors

Jochen Topf ([email protected]), Dane Springmeyer ([email protected])

vtzero's Issues

-Wtautological-constant-compare with clang++ 6.x

I built with latest clang++ 6.x and seeing a new warning:

cd /Users/dane/projects/vtzero/build/examples && /Users/dane/.mason/mason_packages/osx-x86_64/clang++/6.0.0/bin/clang++   -I/Users/dane/projects/vtzero/include -isystem /Users/dane/projects/protozero/include  -fsanitize=address,undefined,integer -fno-sanitize-recover=all -fno-omit-frame-pointer -O3 -DNDEBUG   -std=c++11 -Wall -Wextra -pedantic -Wsign-compare -Wconversion -Werror -o CMakeFiles/vtzero-filter.dir/vtzero-filter.cpp.o -c /Users/dane/projects/vtzero/examples/vtzero-filter.cpp
/Users/dane/projects/vtzero/examples/vtzero-filter.cpp:85:26: error: comparison 'const long' > 9223372036854775807 is always false
      [-Werror,-Wtautological-constant-compare]
        if (id < 0 || id > std::numeric_limits<long>::max()) { // NOLINT clang-tidy: google-runtime-int
                      ~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[3]: *** [examples/CMakeFiles/vtzero-filter.dir/vtzero-filter.cpp.o] Error 1
make[2]: *** [examples/CMakeFiles/vtzero-filter.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [release] Error 2

To replicate:

cd vtzero
git clone [email protected]:mapbox/mason.git
./mason/mason install clang++ 6.0.0
export CXX=$(./mason/mason prefix clang++ 6.0.0)/bin/clang++
mkdir build && cd build && cmake ../ && make

Accept linestring geometries containing a ClosePath command

Refs mapbox/mapbox-gl-native#11579
Fixture in mvt-fixtures: mapbox/mvt-fixtures#94

It looks like mapbox/vector-tile effectively just reads ClosePath as MoveTo(first_point_in_this_line). I suppose if we want mapbox-gl-native to behave exactly the same way after moving from vector-tile to vtzero, we'd want to do the same thing here, but I'm not sure if that's necessary. If, for example, simply ignoring the ClosePath command made more sense from vtzero's point of view, mapbox-gl-native would be able to be backwards compatible with a slight degradation that, IMO, would be acceptable.

Build broken with gcc>=11

Building with gcc>=11 results in:

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -I./vendor/vtzero/include -I./vendor/protozero/include -I/usr/include/python3.9 -c vtzero/tile.cpp -o build/temp.linux-x86_64-3.9/vtzero/tile.o -O2 -std=c++11
In file included from ./vendor/vtzero/include/vtzero/property_value.hpp:20,
from ./vendor/vtzero/include/vtzero/property.hpp:19,
from ./vendor/vtzero/include/vtzero/feature.hpp:20,
from ./vendor/vtzero/include/vtzero/layer.hpp:20,
from ./vendor/vtzero/include/vtzero/vector_tile.hpp:20,
from vtzero/tile.cpp:654:
./vendor/vtzero/include/vtzero/types.hpp:286:52: error: ‘numeric_limits’ is not a member of ‘std’
286 | static const uint32_t invalid_value = std::numeric_limits<uint32_t>::max();
| ^~~~~~~~~~~~~~
./vendor/vtzero/include/vtzero/types.hpp:286:75: error: expected primary-expression before ‘>’ token
286 | static const uint32_t invalid_value = std::numeric_limits<uint32_t>::max();
| ^
./vendor/vtzero/include/vtzero/types.hpp:286:78: error: ‘::max’ has not been declared; did you mean ‘std::max’?
286 | static const uint32_t invalid_value = std::numeric_limits<uint32_t>::max();
| ^~~
| std::max

This is because of https://www.gnu.org/software/gcc/gcc-11/porting_to.html:
Header dependency changes

Some C++ Standard Library headers have been changed to no longer include other headers that they do need to depend on. As such, C++ programs that used standard library components without including the right headers will no longer compile.

The following headers are used less widely in libstdc++ and may need to be included explicitly when compiled with GCC 11:

**<limits> (for std::numeric_limits)**
<memory> (for std::unique_ptr, std::shared_ptr etc.)
<utility> (for std::pair, std::tuple_size, std::index_sequence etc.)

Error reporting

Vtzero uses exceptions in many cases to report invalid vector tile data. Those exceptions contain more or less detailed descriptions of what went wrong. What they don't tell you is where something went wrong. You'll get an exception and don't know which feature in which layer this is from. The user code can, of course, figure this out, because it was probably iterating over the layers and features and so it can tell you. But that means extra effort in the user code and you'll have to catch the exceptions and add this information somehow and then rethrow it or display it.

We could add code in vtzero to track the layer and feature index in the vector tile and add it to the exceptions. But this, of course, has some overhead. Is this something we want? Maybe in most cases where an error occurs we just want to ignore everything anyway and don't need detailed reports? Or maybe this is something so useful, we want to have it built in?

Opinions?

@springmeyer @flippmoke

Docs

The vtzero library is reaching stability. @joto's next steps are to work on more documentation. Will update this ticket as progress is made.

Apparent error in Y coordinate locations.

I have run the program vtzero-create and opened the resulting tile in qgis. The y-coordinates appear to be shifted by 4096. In the polygon layer (shown below), the top left corner is at (0, 4096) and the bottom right corner is at (10, 4086). Note the coordinate in the red ellipse.

From the code, I thought these should be (0, 10) and (10, 0) respectively.

And the JSON is (decoded with mapbox-vector-tile):

{
    "points": {
        "extent": 4096,
        "version": 2,
        "features": [
            {
                "geometry": {
                    "type": "Point",
                    "coordinates": [
                        20,
                        4076
                    ]
                },
                "properties": {
                    "some": "attr"
                },
                "id": 2,
                "type": "Feature"
            },
            {
                "geometry": {
                    "type": "Point",
                    "coordinates": [
                        20,
                        4076
                    ]
                },
                "properties": {
                    "some": "attr"
                },
                "id": 3,
                "type": "Feature"
            },
            {
                "geometry": {
                    "type": "Point",
                    "coordinates": [
                        20,
                        4076
                    ]
                },
                "properties": {
                    "some": "otherattr"
                },
                "id": 4,
                "type": "Feature"
            },
            {
                "geometry": {
                    "type": "Point",
                    "coordinates": [
                        20,
                        4076
                    ]
                },
                "properties": {
                    "otherkey": "attr"
                },
                "id": 5,
                "type": "Feature"
            }
        ],
        "type": "FeatureCollection"
    },
    "lines": {
        "extent": 4096,
        "version": 2,
        "features": [
            {
                "geometry": {
                    "type": "MultiLineString",
                    "coordinates": [
                        [
                            [
                                10,
                                4086
                            ],
                            [
                                10,
                                4076
                            ],
                            [
                                20,
                                4076
                            ]
                        ],
                        [
                            [
                                11,
                                4085
                            ],
                            [
                                12,
                                4083
                            ]
                        ]
                    ]
                },
                "properties": {
                    "highway": "primary",
                    "maxspeed": 50
                },
                "id": 6,
                "type": "Feature"
            }
        ],
        "type": "FeatureCollection"
    },
    "polygons": {
        "extent": 4096,
        "version": 2,
        "features": [
            {
                "geometry": {
                    "type": "Polygon",
                    "coordinates": [
                        [
                            [
                                0,
                                4096
                            ],
                            [
                                10,
                                4096
                            ],
                            [
                                10,
                                4086
                            ],
                            [
                                0,
                                4086
                            ],
                            [
                                0,
                                4096
                            ]
                        ],
                        [
                            [
                                3,
                                4093
                            ],
                            [
                                3,
                                4091
                            ],
                            [
                                5,
                                4091
                            ],
                            [
                                3,
                                4093
                            ]
                        ]
                    ]
                },
                "properties": {
                    "natural": "wood",
                    "number_of_trees": 23402752
                },
                "id": 7,
                "type": "Feature"
            }
        ],
        "type": "FeatureCollection"
    }
}

Am I misunderstanding something or is this a glitch?

I am using GCC 9.4.0, on Ubuntu 20.04, and commit 32c3648 (head of master at the time of submitting this issue).

Improvements for efficiently adding layers via bytes

An important usecase is appending raw bytes of existing messages.

Currently the tile_builder has add_layer_with_data which is close to what is needed. However it only accepts an already created vtzero::layer. Ideally you could pass in a vtzero::data_view (representing a layer) directly if you have one (without needing to createa a vtzero::layer).

Also, let's think about efficiently adding features to layers via bytes. This usecase might sound odd because normally you need to build up layer keys/values based on feature properties. But in certain high performance cases one might want to be able to filter certain features on a layer and keep others (without re-encoding the layer key/values). So, to efficiently we need to be able to append the raw bytes of features to layers.

/cc @joto

Document ownership relation between layer and feature

It looks to me like vtzero::feature depends on having a pointer to its vtzero::layer object (specifically for getting property data):

vtzero/include/vtzero/layer.hpp

Lines 388 to 396 in 62d2640

	inline property feature::next_property() {
	const auto idxs = next_property_indexes();
	property p{};
	if (idxs.valid()) {
	p = {m_layer->key(idxs.key()),
	m_layer->value(idxs.value())};
	}
	return p;
	}

However, this requirement isn’t enforced--it's possible, for instance, to do:

vtzero::feature getFeature(vtzero::tile tile, std::string layer, uint64_t id) {
  return tile.get_layer_by_name(layer).get_feature_by_id(id);
}

and end up with a feature whose owning layer has been destroyed.

Since presumably the overhead of enforcing this with a shared_ptr would be undesirable, would it make sense to at least make a note of this in the documentation for feature?

VT3 support

This ticket tracks the development of support for the VT3 format. (https://github.com/mapbox/vector-tile-spec/tree/v3.0-development).

Branch vt3

Contains support for:

• string ids
• inline attributes including new array and map nested attributes (mapbox/vector-tile-spec#121)
• number lists
• 3d geometry support
• x/y/zoom in layer
• geometric attributes
• spline support

@ericfischer @flippmoke

v1.0.2 release

I noticed that 6d3dd89 is fixed only in master. Should we do a new v1.0.2 release now @joto?

"Tag" vs. "Property"

The vector tile spec uses "tag" and "property" somewhat interchangably. Which is the preferred term?

/cc @flippmoke, @springmeyer, @mapsam

Performance difference with two ways of iterating feature properties

@joto the docs nicely explain how you can either use feature.next_property() or feature.for_each_property([&](property&& p) {.... But they do not mention any performance considerations? Is the performance/complexity equal and these interfaces are purely two ways of doing the same thing? Or should you choose, if you have the ability, one or another if you want the fastest?

Either way it would be great to add mention of whether their perf is equal or potentially different to the docs (https://github.com/mapbox/vtzero/blob/master/doc/tutorial.md#the-feature)

refs

vtzero/include/vtzero/layer.hpp

Lines 388 to 408 in 2ba9ed7

	inline property feature::next_property() {
	const auto idxs = next_property_indexes();
	return idxs.valid() ? property{m_layer->key(idxs.key()),
	m_layer->value(idxs.value())}
	: property{};
	}
	template <typename TFunc>
	bool feature::for_each_property(TFunc&& func) const {
	vtzero_assert(valid());
	for (auto it = m_properties.begin(); it != m_properties.end();) {
	const uint32_t ki = *it++;
	const uint32_t vi = *it++;
	if (!std::forward<TFunc>(func)(property{m_layer->key(ki), m_layer->value(vi)})) {
	return false;
	}
	}
	return true;
	}

/cc @GretaCB

Proper down casting of size (-Wshorten-64-to-32)

..../vtzero/include/vtzero/builder.hpp:618:34: error: implicit conversion loses integer precision: 'size_type'
      (aka 'unsigned long') to 'uint32_t' (aka 'unsigned int') [-Werror,-Wshorten-64-to-32]
            add_points(container.size());

https://github.com/mapbox/vtzero/blob/master/include/vtzero/builder.hpp#L618

Indexes for key/value tables in layers

This issue documents the current state of affairs concerning indexes for key/value tables. Other issues can refer back to this for context.

When building a new layer you need to populate the key/value tables while adding new properties. There are several ways of doing this:

1. Use the builtin index. This is the easiest choice, but performance isn't great. It uses a flat vector with linear search for small numbers of entries or an std::unordered_map when there are more. Some quick benchmarks show that the current value of 20 entries beyond which the implementation switches to std::unordered_map is reasonable.
2. Use the property_mapper. This is the best choice when copying some features (or some properties of some features) from one existing layer, it is more than twice as fast as the builtin index.
3. Use one of the supplied indexes in index.hpp.
4. Do everything yourself. You know the data best that you are adding and can choose the right strategy.

Medium term we should think about a better implementation for (1), but this needs more benchmarks with real data and different implementations to find the best one. Because this is hidden from the user of the library, we can always improve on this later.

It is unlikely that we'll find a much better approach for (2) than the current one. But this is only usable in very specific circumstances.

We can always add to (3), for instance adding vector-based flat maps with linear search.

Builder - Make feature id optional

Currently, the constructor of the different feature builder has an id as a field, this will always set a value of 0 for the feature id, even if no feature id is required. I would propose for the id to be set outside the constructor of the feature builders much like properties.

`add_feature` is broken in v1.0.1

inline void layer_builder::add_feature(const feature& feature) {
        geometry_feature_builder feature_builder{*this};
        if (feature.has_id()) {
            feature_builder.set_id(feature.id());
        }
        feature_builder.set_geometry(feature.geometry());
        feature.for_each_property([&feature_builder](const property& p) {
            feature_builder.add_property(p);
            return true;
        });
    }

When feature_builder goes out of scope its dtor calls rollback() and no feature added
/cc @joto

Features upside down

Hi there

I used your awesome vtzero to create a pbf2geojson converter. It works and is pretty fast, compared to my current python solution. However, all features are rendered upside down. I hope you can help me with that.

Thanks and best regards
Martin

Minimal vector tile C++ decoder and encoder

@joto has created a solid design for new, higher performance and more minimalistic vector tile decoder and encoder as part of mapbox/vector-tile#20. This vtzero repo will be the home for this work.

This implementation will only depend on protozero and will make no assumptions about the property, feature, or geometry storage you will decode into or encode from. Rather it will provide handlers for minimal geometry and property decoding and encoding. The https://github.com/mapbox/vector-tile will be re-written to be a higher level wrapper around vtzero for handling vtzero <-> geometry.hpp, projections, and other APIs that will depend on C++ modules in the mapbox:: namespace.

@joto - let's use this ticket for the initial coordination. The next steps I'm seeing are:

• @springmeyer creates initial skeleton of repo based on hpp-skel from the @mapbox/core-tech team.
• @joto reviews and pings @springmeyer if any of the skeleton needs modified in major ways
• @joto tags latest protozero at 1.5.3 (as I think latest vtzero depends on it)
• @springmeyer packages protozero 1.5.3 in mason
• @springmeyer updates this repo to pull in protozero 1.5.3 - 19dc27e
• @joto moves the vtzero code over from vector-tile to here.

After that there will still be much to do. Like writing tests, benchmarks, and docs (tracked in milestone 1). As well as back-porting fixes we make here that are appropriate for hpp-skel. But we can handle all those things in followup tickets.

Handling different encodings for repeated fields

The protobuf spec says that packed repeated fields must also be parsed if they are not using the packed format and that multiple packed fields are also allowed and should be concatenated. This affects the tags and geometry fields in the features.

The current vtzero code doesn't handle this properly. It is rather unlikely that this will happen in real life, but we have to think about whether we want to allow this. The problem is that this will most likely be more expensive to handle properly. I'll have to think about the details here some more.

Is it possible to add empty layers in the tile_builder

An example mvt here contains layers with no geometries. I am using this as a test tile for creating a writer using vtzero, but I cannot figure out how to add an empty layer.

The use case is because, I think there is a difference between:

• my dataset (a collection of mvts) contains a particular layer but an individual mvt may not have any features for that layer
• my dataset does not contain a layer by that name

Currently, as far as I can understand, vtzero cannot distinguish between the two. Is that correct? Does the spec permit empty layers?

In writing.md there is a statement:

The tile builder is smart enough to not add empty layers, so you can start out with all the layers you might need and if some of them stay empty, they will not be added to the tile when serialize() is called.

Which I assume means that the inability to ignore empty layers is intentional.

cython should be a dependency of vtzero

(env) laptop% pip install vtzero
Collecting vtzero
Using cached https://files.pythonhosted.org/packages/4a/70/d5171240be0bfe4d6ae0002a19c9ee0bd17d1a7670facfad471a023c58ad/vtzero-0.0.1b1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-1wgby62w/vtzero/setup.py", line 5, in
from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-1wgby62w/vtzero/
You are using pip version 19.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I believe this is possible with PEP518

add_property is slow

when add 100 property value, add_property is slow ,how to optimize add_property

Create mvt fixtures for geometry tests

The tests in test/t/test_geometry.cpp test the geometries in isolation by crafting special broken geometries. These should be ported to tests in mvt-fixtures.

Geometry handling: strict vs. non-strict mode

Currently the geometry handling has a run-time setting strict that enables strict mode in which the library does some more tests on the geometry it got from the vector tile.

Originally I thought the user code would enable this for version 2 tiles and disable this for version 1 tiles. But I am not sure this makes sense. If the code using these geometries can handle them in non-strict mode anyway, why the extra checks? And if it can't, we'll always need those checks. Basically what those checks give you is additional postconditions on the geometry and it depends on the users code whether it can handle those postconditions, this is not something that usually depends on run-time information.

Do we actally need this? Does this have to be a run-time setting? Maybe this is something that should be a template parameter.

@flippmoke

Repeated LineTo Commands

While I feel indirectly the 2.x version of the spec is clear that you should not have repeated lineTo commands in the case of polygons/linestrings and that you should just have an increased command count -- there was no such restriction on version 1.

https://github.com/mapbox/vector-tile-spec/tree/master/2.1#4343-linestring-geometry-type

Therefore, I wonder if we should adjust the code the linestring decode and polygon decode to support multiple lineto commands as a possibility.

Tests and documentation for vtzero::key_index/value_index

@joto - I noticed that you are working on a more flexible way to control how the tables are built for keys and values during the building of a new vector tile.

The new interface I notice in https://github.com/mapbox/vtzero/blob/master/include/vtzero/index.hpp looks powerful - the idea of allowing the programmer to control whether a std::map or std::unordered_map (or something else like a sso optimized map - https://github.com/greg7mdp/sho) is used it excellent.

I'd like to put a request in to prioritize this code for documentation and tests because I anticipate that @GretaCB and I will want to use it, and benchmark plugging in different containers.

Currently in our code I'm seeing that the default implementation (which appears to iterate the protobuf data each time a key/value is added?) is a bottleneck. So I'm hoping a hashmap will be faster:

refs

vtzero/include/vtzero/builder_impl.hpp

Lines 66 to 79 in 094cd4b

	static index_value find_in_table(const data_view text, const std::string& data) {
	uint32_t index = 0;
	protozero::pbf_message<detail::pbf_layer> pbf_table{data};
	while (pbf_table.next()) {
	const auto v = pbf_table.get_view();
	if (v == text) {
	return index_value{index};
	}
	++index;
	}
	return index_value{};
	}

Integrate [email protected]

Once the upcoming release of mvt-fixtures is out, we should upgrade vtzero to use them.

layer_builder.rollback() ?

I have a usecase where I create a layer_builder and then add features to it. In the event that no features are added (due to filtering) I'd like to avoid adding this layer to the tile. But by the time I know that no features have been added, the layer has already been added to the tile and queued up to be serialized. How best can this be best solved? Could the code here check for whether features have been added and if so skip calling ->build()?

/cc @joto

Unable to parse tile that is likely valid

Here is a tile that I think is valid but cannot be parsed with vtzero: https://www.dropbox.com/s/zxj4vdklom9gav2/13_1310_3166.pbf?dl=0 (link will expire in 7 days).

Perhaps it is invalid. Or perhaps vtzero is not doing something right. I'm not sure. @joto - can you investigate?

Parsing throws like:

(lldb) thread backtrace all
* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x00007fffce6d2d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fffce7c0457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fffce638420 libsystem_c.dylib`abort + 129
    frame #3: 0x00007fffcd18b94a libc++abi.dylib`abort_message + 266
    frame #4: 0x00007fffcd1b0c17 libc++abi.dylib`default_terminate_handler() + 243
    frame #5: 0x00007fffcdcc0713 libobjc.A.dylib`_objc_terminate() + 124
    frame #6: 0x00007fffcd1add49 libc++abi.dylib`std::__terminate(void (*)()) + 8
    frame #7: 0x00007fffcd1ad7be libc++abi.dylib`__cxa_throw + 121
    frame #8: 0x00000001004cf7b7 vtzero-show`protozero::pbf_reader::next(this=0x00007fff5f735300) at pbf_reader.hpp:295
    frame #9: 0x00000001004d1093 vtzero-show`protozero::pbf_reader::next(this=0x00007fff5f735300, next_tag=3, type=length_delimited) at pbf_reader.hpp:368
    frame #10: 0x00000001004d31d1 vtzero-show`protozero::pbf_message<vtzero::detail::pbf_tile>::next(this=0x00007fff5f735300, next_tag=layers, type=length_delimited) at pbf_message.hpp:164
    frame #11: 0x00000001004d3133 vtzero-show`vtzero::tile_iterator::next(this=0x00007fff5f735300) at vector_tile.hpp:27
    frame #12: 0x00000001004d30fe vtzero-show`vtzero::tile_iterator::tile_iterator(this=0x00007fff5f735300, tile_data=0x00007fff5f735330) at vector_tile.hpp:56
    frame #13: 0x00000001004d308d vtzero-show`vtzero::tile_iterator::tile_iterator(this=0x00007fff5f735300, tile_data=0x00007fff5f735330) at vector_tile.hpp:55
    frame #14: 0x00000001004cda9c vtzero-show`vtzero::vector_tile::begin(this=0x00007fff5f735330) const at vector_tile.hpp:140
    frame #15: 0x00000001004cd6c5 vtzero-show`main(argc=2, argv=0x00007fff5f735440) at vtzero-show.cpp:237
    frame #16: 0x00007fffce5a4235 libdyld.dylib`start + 1

feature_builder.add_tags(uint32_t key_idx, uint32_t value_idx)

It appears not possible currently to add tag elements directly to the feature_builder_base class as uint32_t.

Would it be possible to add them directly to bypass the expense of looking up the uint32_t idx via the layer's std::map in the case where the correct index is already known?

/cc @joto

Testing goals

As vtzero reaches stability @joto has expressed planning on working on more tests. @joto can you share here what your % coverage goals are and what major needs remain for tests?

Do we care about sanitizing unsigned integer overflow on travis?

The clang docs note:

-fsanitize=unsigned-integer-overflow: Unsigned integer overflows. Note that unlike signed integer overflow, unsigned integer is not undefined behavior. However, while it has well-defined semantics, it is often unintentional, so UBSan offers to catch it.

By reading the docs closely (confirmed by looking at https://github.com/llvm-mirror/clang/blob/master/include/clang/Basic/Sanitizers.def) it looks like unsigned-integer-overflow is enabled with the -fsanitize=integer group (and with -fsanitize=unsigned-integer-overflow specifically) but not by the -fsanitize=undefined group.

@joto I noticed 8320b87, which I think means we will no longer be testing -fsanitize=unsigned-integer-overflow. Should we add that back or are we fine not worrying about unsigned integer overflow (since it is defined)?

Remove extra "add_points"

As I was reviewing the code in builder.hpp I noticed that point_feature_builder has several repeated ways to add multiple points from a container type structure:

• https://github.com/mapbox/vtzero/blob/master/include/vtzero/builder.hpp#L404-L411
• https://github.com/mapbox/vtzero/blob/master/include/vtzero/builder.hpp#L427-L438
• https://github.com/mapbox/vtzero/blob/master/include/vtzero/builder.hpp#L452-L461

The first example requires a call to std::distance however, which I feel people might utilize with out knowing the other methods exist. Should we limit the number of methods here to encourage people to use the best method?

Compilation failure on M1 mac

Does this project support ARM? Specifically the M1 mac's. I'm trying to compile this on an M1 mac and I'm getting a unrecognized instruction mnemonic error and from a quick web search it appears that this indicates that the issue is caused by ARM.

Here's the full compilation log:

jason@Jasons-Work-MacBook-Air-8 ~/d/f/v/build (master)> cmake ..
CMake Deprecation Warning at CMakeLists.txt:9 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Found Boost: /opt/homebrew/lib/cmake/Boost-1.78.0/BoostConfig.cmake (found version "1.78.0")
-- Looking for clang-tidy
-- Looking for clang-tidy - not found
--   Build target 'clang-tidy' will not be available.
-- Looking for cppcheck
-- Looking for cppcheck - not found
--   Build target 'cppcheck' will not be available.
-- Looking for iwyu
-- Looking for iwyu - not found
--   Build target 'iwyu' will not be available.
-- Configuring documentation
-- Looking for doxygen
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Looking for doxygen - not found
--   Disabled making of documentation.
-- Configuring documentation - done
-- Adding ext test: mapbox-streets-v6-14-8714-8017.mvt
-- Boost library found: enable testing with boost::variant
-- Found test fixtures. Enabled mvt fixture tests.
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jason/dev/forks/vtzero/build
jason@Jasons-Work-MacBook-Air-8 ~/d/f/v/build (master)> make
[  2%] Building CXX object examples/CMakeFiles/vtzero-check.dir/vtzero-check.cpp.o
[  4%] Building CXX object examples/CMakeFiles/vtzero-check.dir/utils.cpp.o
[  6%] Linking CXX executable vtzero-check
[  6%] Built target vtzero-check
[  9%] Building CXX object examples/CMakeFiles/vtzero-create.dir/vtzero-create.cpp.o
[ 11%] Building CXX object examples/CMakeFiles/vtzero-create.dir/utils.cpp.o
[ 13%] Linking CXX executable vtzero-create
[ 13%] Built target vtzero-create
[ 15%] Building CXX object examples/CMakeFiles/vtzero-encode-geom.dir/vtzero-encode-geom.cpp.o
[ 18%] Building CXX object examples/CMakeFiles/vtzero-encode-geom.dir/utils.cpp.o
[ 20%] Linking CXX executable vtzero-encode-geom
[ 20%] Built target vtzero-encode-geom
[ 22%] Building CXX object examples/CMakeFiles/vtzero-stats.dir/vtzero-stats.cpp.o
[ 25%] Building CXX object examples/CMakeFiles/vtzero-stats.dir/utils.cpp.o
[ 27%] Linking CXX executable vtzero-stats
[ 27%] Built target vtzero-stats
[ 29%] Building CXX object examples/CMakeFiles/vtzero-streets.dir/vtzero-streets.cpp.o
[ 31%] Building CXX object examples/CMakeFiles/vtzero-streets.dir/utils.cpp.o
[ 34%] Linking CXX executable vtzero-streets
[ 34%] Built target vtzero-streets
[ 36%] Building CXX object examples/CMakeFiles/vtzero-filter.dir/vtzero-filter.cpp.o
[ 38%] Building CXX object examples/CMakeFiles/vtzero-filter.dir/utils.cpp.o
[ 40%] Linking CXX executable vtzero-filter
[ 40%] Built target vtzero-filter
[ 43%] Building CXX object examples/CMakeFiles/vtzero-show.dir/vtzero-show.cpp.o
[ 45%] Building CXX object examples/CMakeFiles/vtzero-show.dir/utils.cpp.o
[ 47%] Linking CXX executable vtzero-show
[ 47%] Built target vtzero-show
[ 50%] Building CXX object test/CMakeFiles/unit-tests.dir/test_main.cpp.o
In file included from /Users/jason/dev/forks/vtzero/test/test_main.cpp:2:
/Users/jason/dev/forks/vtzero/test/catch/catch.hpp:8234:13: error: unrecognized instruction mnemonic, did you mean: bit, cnt, hint, ins, not?
            CATCH_BREAK_INTO_DEBUGGER();
            ^
/Users/jason/dev/forks/vtzero/test/catch/catch.hpp:7945:83: note: expanded from macro 'CATCH_BREAK_INTO_DEBUGGER'
        #define CATCH_BREAK_INTO_DEBUGGER() []{ if( Catch::isDebuggerActive() ) { CATCH_TRAP(); } }()
                                                                                  ^
/Users/jason/dev/forks/vtzero/test/catch/catch.hpp:7910:34: note: expanded from macro 'CATCH_TRAP'
    #define CATCH_TRAP() __asm__("int $3\n" : : ) /* NOLINT */
                                 ^
<inline asm>:1:2: note: instantiated into assembly here
        int $3
        ^
1 error generated.
make[2]: *** [test/CMakeFiles/unit-tests.dir/test_main.cpp.o] Error 1
make[1]: *** [test/CMakeFiles/unit-tests.dir/all] Error 2
make: *** [all] Error 2

Getopt library on Windows

Building/Testing of some of the examples is disabled on Windows because of the missing getopt library. We should figure something out here.

See also mapbox/protozero#84.

Mascot?

What about a friendly mascot for vtzero? Seeing the delightful mvt-fixtures crew at https://github.com/mapbox/mvt-fixtures makes me wonder if a great vtzero mascot could be one of those little creatures with a proud 0 in the middle (hole with cross) and some arms that are strongly holding up a globe? Or otherwise in a position as if it is holding something up (to get at the fact that vtzero is foundational)?

Thoughts?

/cc @mapsam @mapbox/core-tech @joto

Question: What coordinate system should I be writing coordinates in?

Hi there,

I'm experimenting with vtzero for creating MVT files and I can't quite grasp from the examples or documentation as to what coordinate system I am meant to be writing geometry coordinates in.

Let's assume I am trying to encode a polygon geometry whose coordinates are in EPSG:4326 (latitude/longitude). Do I:

1. Write these coordinate as-is?
2. Transform/clip the geometry to a 4096x4096 "pixel" screen space of the applicable tile such that the coordinates written are in the range between (0, 0) and (4095, 4095)? I gathered 4096 is the default tile size
3. Something else?

Asserts in set_point

There are three asserts in the set_point() functions of the linestring_feature_builder and polygon_feature_builder:

1. https://github.com/mapbox/vtzero/blob/master/include/vtzero/builder.hpp#L812
2. https://github.com/mapbox/vtzero/blob/master/include/vtzero/builder.hpp#L1014
3. https://github.com/mapbox/vtzero/blob/master/include/vtzero/builder.hpp#L1018

(1) and (3) are used to enforce the spec rule ("For any pair of (dX, dY) the dX and dY MUST NOT both be 0") (2) is used to make sure the last point is the same as the first point so the ClosePath works correctly.

I am not sure what the right behaviour in these cases is. If we leave them as asserts, the calling code has to make sure these conditions hold. If we throw an exception instead, this makes it easier for the calling code. On the other hand, it is an extra check that some code might not need (code that is already sure the geometries are okay) that costs time and resources (the m_first_point member would not be needed without check (2)).

Opinions? @springmeyer @flippmoke

Handling of coordinate overflow in geometries.

Coordinates in vector tile geometries are stored in the vector tiles as delta-encoded and zigzag-encoded 32bit varints. There are at least two places where an overflow can happen:

• If the varint encodes something larger than would fit in a 32bit int, it is silently truncated by protozero when reading (because vtzero asks protozero for a 32bit, not a 64bit int). It is not possible to generate data like this using any normal protobuf library.
• If the delta encoding generates something larger than 32bit. Because the delta-encoding is symmetric it is not possible to generate something like this using any normal vector tile code.

How should we handle these overflows? I put in a check recently for the 2nd one, but now think that this probably is not necessary. We can just ignore those overflows, because

1. They will never happen with real data. (Coordinates are usually between 0 and the extent (4096) plus some values for a buffer around the tiles. Nowhere near the limits of an int32.)
2. If somebody crafts these specially, we don't care what the resulting geometries look like.

The only issue here is that it might be possible to generate geometries violating some constraint (particularly the no-zero-lengt-segment rule) that would confuse later users of the data. But it should be possible to change the code to check for that.

@springmeyer, @flippmoke any opinions?