mapbox / vector-tile

C++14 library for decoding Mapbox Vector Tiles

Vector Tile Library

C++14 library for decoding Mapbox Vector Tiles.

Depends

• C++14 compiler
• protozero
• variant
• geometry

Building

Call

git submodule init
git submodule update

to install test fixtures from an external git repository.

To install all dependencies and build the tests in this repo do:

make test

To bundle the demo program do:

make demo

This copies all the includes into the demo/include folder such that the demo can be build like:

make -C demo/

Or also like:

cd demo
make

Who is using vector-tile?

These are the applications targeted to upgrade to this library:

• Mapbox GL Native
• Mapnik Vector Tile
• Tippecanoe

vector-tile's Issues

getValue does not handle layer with key duplicates

When decoding a layer with key duplicates, getValue throws "feature referenced out of range key". A layer SHOULD NOT contain key duplicates by the spec, but tiles generated by Postgis ST_AsMVT can contain them.

The pull request #53 fixes that issue. I'll add a pull request for another possible fix.

Possible optimization in v2 branch

The geometry decoding in https://github.com/mapbox/vector-tile/blob/2.x/include/mapbox/vector_tile/handlers.hpp works by first creating "multi" geometries of the given type. If later the geometry turns out to only contain a single point, linestring, or polygon, the first geometry of the "multi" is returned. In practice most geometries are probably not multi, so creating this extra indirection (which internally is a std::vector) and then discarding it could have measurble performance impact.

Use std::unordered_map<std::string, ...>

It's typically faster.

Add Decoding to WGS 84 / Web Mercator

Add the ability to take data from a vector tile and return features in wgs84 and web mercator.

Deprecate v1.x / merge 2.x into master

Currently the only (known) application depending on the v1 version of this library is MBGL. Once MBGL moves to vtzero (mapbox/mapbox-gl-native#11489) I propose we move forward with v2.x here by:

• merging v2.x branch (https://github.com/mapbox/vector-tile/tree/2.x) into master
• adding a deprecation note in the readme that using v1.x is no longer recommended for us
• adding a section of the readme that describes when you'd use vtzero vs mapbox/vector-tile
• consider renaming this repo to mapbox/libvectortile since mapbox/vector-tile is so easily confused given its generic name

Sound like a good plan? @flippmoke @kkaefer @anandthakker @joto

Context: v1.x of this library was never well tested. Vtzero is extremely well tested. The v2.x branch is now based on top of vtzero and is therefore a higher level, more robust replacement.

travis tests

Would be great to tests up and running on travis. Going to start by pulling from hpp-skel.

cc @springmeyer @flippmoke

publish to mason

Let's get vector-tile published to mason so we can include it in other projects (instead of copy-pasting).

https://github.com/mapbox/hpp-skel#deploy-to-mason-packing

cc @springmeyer

Phase 2 decoding

Plan:

• Write decoding utility functions to unpack vector tiles, in C++, into the feature+geometry structure of https://github.com/mapbox/geometry.hpp
  • Unpacks into tile coordinates, no assumed projection.
• Primary goal is to have this library be used in Mapbox GL native
• Secondary goal is to have this library be used in mapnik-vector-tile and tippecanoe.
• Depends on #2, protozero, geometry.hpp, evilmvt fixtures, and happymvt fixtures.
  • Uses latest protozero features: -DPROTOZERO_STRICT_API
• Only supports v2 vector tiles (not v1)
• Will be packaged in mason, able to be pulled back into mapbox-gl-native like geometry.hpp
• Will be generic code suitable to be plugged back into/replace parts of:
  • https://github.com/mapbox/mapbox-gl-native/blob/master/src/mbgl/tile/vector_tile.cpp
  • https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_geometry_decoder.ipp
  • https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_featureset_pbf.ipp
  • https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_datasource_pbf.ipp

Longer term questions:

• For mapnik-vector-tile to upgrade to this lib, Mapnik would likely need the ability to support non-projected tile coordinate data

Library is not parsing vector tiles received from mapbox streets service

The library is parsing all .mvt vector tiles that come together w the library as test data. However, when I try to feed .mvt that I received from mapbox streets service, the library fails to parse it. Attached is .mvt that I received in response to this request:
https://api.mapbox.com/v4/mapbox.mapbox-streets-v8/14/4681/6261.mvt?access_token=. The returned .mvt looks quite small (considering the fact that the requested area is the center of Moscow) but this seems to the the valid response.
0_0.zip

Compile demo in Microsoft Visual Studio Professional 2017

Hi! I have some problems with compilation "demo" in Microsoft Visual Studio Professional 2017.

I can't to compile the function "static mapbox::geometry::value parseValue(protozero::data_view const& value_view)" in the module vector_tile.hpp with message "error C2679: binary '=': no operator found which takes a right-hand operand of type 'std::string' (or there is no acceptable conversion)" and so on (see image please).

Can any body help me please to modify sources for compile them with Microsoft Visual Studio Professional 2017?

With best regadrs, Andrey Chernyshov

More unit tests of decoder

Currently the decoder is only being tested against one sample tile (

vector-tile/test/unit/vector_tile.test.cpp

Lines 55 to 81 in 7437763

	TEST_CASE( "Read tile" ) {
	std::string buffer = open_tile("test/mvt-fixtures/fixtures/valid/Feature-single-point.mvt");
	mapbox::vector_tile::buffer tile(buffer);
	const auto layer_names = tile.layerNames();
	REQUIRE(layer_names.size() == 1);
	REQUIRE(layer_names[0] == "layer_name");
	auto layer_ptr = tile.getLayer("layer_name");
	REQUIRE(layer_ptr != nullptr);
	const mapbox::vector_tile::layer & layer = *layer_ptr;
	REQUIRE(layer.featureCount() == 1);
	REQUIRE(layer.getName() == "layer_name");
	auto feature_ptr = layer.getFeature(0);
	const mapbox::vector_tile::feature & feature = * feature_ptr;
	auto feature_id = feature.getID();
	REQUIRE(feature_id);
	REQUIRE(feature_id->is<uint64_t>());
	REQUIRE(feature_id->get<uint64_t>() == 123ull);
	REQUIRE(feature.getType() == mapbox::vector_tile::GeomType::POINT);
	auto props = feature.getProperties();
	auto val = props["hello"];
	REQUIRE(val.is<std::string>());
	REQUIRE(val.get<std::string>() == "world");
	auto opt_val = feature.getValue("hello");
	REQUIRE(opt_val);
	REQUIRE(opt_val->is<std::string>());
	REQUIRE(opt_val->get<std::string>() == "world");
	}

). Next we need to build out the tests to ensure the decoder is working against all valid and invalid tiles from https://github.com/mapbox/mvt-fixtures

IDs ignored

IDs < 0 and IDs not of type int64_t or uint64_t are simply ignored here:
https://github.com/mapbox/vector-tile/blob/2.x/include/mapbox/vector_tile/builder.hpp#L48-L59

Is this correct? If yes, I think it needs to be documented.

Add Overzooming

It is a common operation to overzoom from a lower zoom level tile to a higher zoom level tile. Effectively making a vector tile from a higher level vector tile. This is something that is commonly done with mapnik-vector-tile. This operation should be added to the vector tile library.

Coordination

The goal of this library is to start sharing core code for working with Mapbox Vector Tiles (https://github.com/mapbox/vector-tile-spec/) across key C++ codebases implementing the specification. The focus will be on Mapbox maintained code, but this library should be easily pluggable anywhere.

This ticket stands to track broadly:

• the upstream needs for this library
• the downstream consumers this library is targeting.

Upstream

• Will definitely depend on protozero 1.x
  • Team to coordinate with: @joto
• Will likely depend on mapbox::variant
  • Team to coordinate with: @artemp
• Will likely depend on a first 1.x release of https://github.com/mapbox/geometry.hpp
  • Team to coordinate with: @artemp @jfirebaugh

Downstream

• tippecanoe:
  • Relevant files:
    • https://github.com/mapbox/tippecanoe/blob/master/mvt.hh
    • https://github.com/mapbox/tippecanoe/blob/master/mvt.cc
  • Team to coordinate with: @ericfischer
• OSRM (https://github.com/Project-OSRM/osrm-backend)
  • Relevant files:
    • https://github.com/Project-OSRM/osrm-backend/blob/master/include/util/vector_tile.hpp
    • https://github.com/Project-OSRM/osrm-backend/blob/master/src/engine/plugins/tile.cpp
  • Team to coordinate with: @lbud @danpat
• Mapbox GL:
  • Relevant files:
    • https://github.com/mapbox/mapbox-gl-native/blob/master/src/mbgl/tile/vector_tile.hpp
  • Team to coordinate with: @kkaefer
• mapnik-vector-tile (https://github.com/mapbox/mapnik-vector-tile)
  • Relevant files:
    • https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_config.hpp
  • Team to coordinate with: @mapsam @jakepruitt @flippmoke

Add Encoding from WGS 84 / Web Merc

Add the ability to perform full encoding of geometry into a tile. This will include the following steps:

• Scaling to Zoom Level
• Data Simplification
• Clipping Data to Tile (with or with out buffer)
• Geometry Correction (should be part of clipping with Wagyu for polygons).
• Encode to buffer

The input for this system should be our geometry.hpp Features, but tooling should be designed in such a way that partial steps could be selected and used by outside libraries if required (to add flexibility).

Build+test setup basics

This lib will depend on:

• mason to install deps
  • protozero
  • geometry.hpp
• test fixtures from https://github.com/mapbox/evilmvt

TODO:

• Modify Makefile to pull down mason deps. We can learn from https://github.com/mapbox/timtam/blob/master/Makefile and https://github.com/mapbox/wagyu/blob/master/Makefile
• Add protozero and geometry.hpp as deps
• Add a build step to pull in the evilmvt fixtures (https://github.com/mapbox/evilmvt/tree/master/fixtures). Should we vendor these or pull in dynamically?

include_dirs so we can build using require

Let's take a cue from Nan and have a include_dirs.js file that allows compliers to use the following syntax when including libs and headers:

'<!(node -e \'require("hpp-skel")\')'

This is super helpful when we run into multiple versions of header files when things are installed with npm.

v2 values of some type are ignored

https://github.com/mapbox/vector-tile/blob/2.x/include/mapbox/vector_tile/builder.hpp#L96-L99

Values of type null_value_t, mapbox::util::recursive_wrapper<std::vector<value>>, and mapbox::util::recursive_wrapper<std::unordered_map<std::string, value>>> are simply ignored here. If this is correct, it needs to be documented.

README has incomplete build instructions.

When, following the README, I type make test, I get the message: Please run 'make release' or 'make debug' first

When, again following the README, calling make demo, I get the message: make: Nothing to be done for 'demo'. The build (instructions) inside the demo directory also don't work.

2.x branch optimization

I noticed that layer_map<CoordinateType> decode_tile(std::string const& buffer) at

vector-tile/include/mapbox/vector_tile.hpp

Lines 62 to 84 in 95c240b

	layer_map<CoordinateType> decode_tile(std::string const& buffer)
	{
	layer_map<CoordinateType> m;
	vtzero::vector_tile tile(buffer);
	while (auto layer = tile.next_layer())
	{
	mapbox::feature::feature_collection<CoordinateType> fc;
	while (auto feature = layer.next_feature())
	{
	auto f = extract_feature<CoordinateType>(feature);
	if (!f.geometry.template is<mapbox::geometry::empty>())
	{
	fc.push_back(f);
	}
	}
	if (!fc.empty())
	{
	m.emplace(std::string(layer.name()), std::move(fc));
	}
	}
	return m;
	}

does not leverage pre-allocation optimizations. We should likely do:

diff --git a/include/mapbox/vector_tile.hpp b/include/mapbox/vector_tile.hpp
index c61640b..0e8c672 100644
--- a/include/mapbox/vector_tile.hpp
+++ b/include/mapbox/vector_tile.hpp
@@ -66,12 +66,13 @@ layer_map<CoordinateType> decode_tile(std::string const& buffer)
     while (auto layer = tile.next_layer())
     {
         mapbox::feature::feature_collection<CoordinateType> fc;
+        fc.reserve(layer.num_features());
         while (auto feature = layer.next_feature())
         {
             auto f = extract_feature<CoordinateType>(feature);
             if (!f.geometry.template is<mapbox::geometry::empty>())
             {
-                fc.push_back(f);
+                fc.push_back(std::move(f));
             }
         }

/cc @flippmoke to review and apply if this looks good.

Enable coverage reporting with codecov.io

Phase 1 / v1.x release

I propose the v1.x release focus exclusively on a common set of headers with enumerations for Mapbox Vector Tile types. See #1 for how this could reduce duplication: e.g. between https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_config.hpp and https://github.com/Project-OSRM/osrm-backend/blob/master/include/util/vector_tile.hpp.

Further phases (2.x and 3.x) would focus on adding decoding and encoding support.

Benchmark script for decoder

The decoder implementation is highly optimized and uses lazy decoding/protozero to avoid high memory usage. But we should write benchmark scripts that can be easily used to catch performance regressions.

Future development

Development on this library is currently paused.

This is because the efforts of @mapbox/core-tech are going towards building out https://github.com/mapbox/vtzero: a more minimalistic, flexible, and efficient implementation (compared to the current API here in master).

The next steps are to work on https://github.com/mapbox/vtzero/milestone/1. Once that is done, we will recommence development on this library by rebranching from hpp-skel.

The goals of the future design will be to add high level APIs on top of vtzero that deal with:

phase 1:

• Decoding geometries into geometry.hpp (https://github.com/mapbox/geometry.hpp)
• Decoding properties into variant.hpp (https://github.com/mapbox/variant)

phase 2:

(via spatial-algorithms)

• Handling projection and other coordinate transformations into tile coords
• Clipping and simplification (https://github.com/mapbox/wagyu)
• Compression (https://github.com/mapbox/gzip-hpp)

/cc @mapbox/core-tech @mapbox/gl-core @mapbox/directions @kkaefer @jfirebaugh @anandthakker @ericfischer

Phase 3: encoding

This issue is to track overall discussion on the next phase of development for this library: encoding.

The previous phases were Phase 1 (enumerations) and Phase 2 (decoding) (landed in mapbox-gl-native in mapbox/mapbox-gl-native#9312 🎉 ).

The primary goals for the encoding implementation are:

• High performance / minimal overhead / low to zero allocation
• Headers are modular and able to be used in a zero cost way (you don't get any extra dependencies than absolutely required for a given header - e.g. no dependency on geometry.hpp or variant unless the header absolutely needs it)
• API is low-level like protozero. Downstream developers will be able to easily abstract on top of it be be able to integrate into libraries like mapnik-vector-tile / tippecanoe.

Roles

• @joto will lead development of design over next 3 weeks (July 3-21)
• @flippmoke will support, test, advise on viability of API for integration downstream

/cc @mapbox/core-tech

Create Benchmarks For 2.x

We do not have extensive benchmarks around the encoding and decoding of tiles in the 2.x branch currently. The following are good benchmarks that we could have:

Decode:

• point
• multipoint
• linestring
• multi linestring
• polygon
• multi polygon

Encode:

• point
• multipoint
• linestring
• multi linestring
• polygon
• multi polygon

/cc @GretaCB

Compositing of vector tile buffers

Add the ability to composite multiple vector tiles into a single vector tile. This should be similar to the compositing functionality of mapnik-vector-tile.

Take small objects by value

The functions in https://github.com/mapbox/vector-tile/blob/2.x/include/mapbox/vector_tile/builder.hpp#L72-L87 should probably take their parameters as values.