isabella232 / vtcomposite

vtzero-based vector tile compositor (supporting overzooming)

Overview

npm install @mapbox/vtcomposite

Compositing is a tool to combine multiple vector tiles into a single tile. For a more in depth explanation about how compositing works, see the tutorial.

vtcomposite is made possible with node-cpp-skel, vtzero, geometry.hpp, spatial-algorithms, gzip-hpp and boost geometry.

API

Table of Contents

• vtcomposite
  • Parameters
  • Examples
  • Tutorial

vtcomposite

Parameters

• tiles Array<Object> an array of tile objects with buffer, z, x, and y values
• zxy Array<Object> z, x, and y values of a map request
• options Object?
  • options.compress Boolean a boolean value indicating whether or not to return a compressed buffer. Default is to return a uncompressed buffer. (optional, default false)
  • options.buffer_size Number the buffer size of a tile, indicating the tile extent that should be composited and/or clipped. Default is buffer_size=0. (optional, default 0)

Examples

const vtcomposite = require('@mapbox/vtcomposite');
const fs = require('fs');

const tiles = [
  { buffer: fs.readFileSync('./path/to/tile.mvt'), z: 15, x: 5238, y: 12666 },
  { buffer: fs.readFileSync('./path/to/tile.mvt'), z: 15, x: 5238, y: 12666 }
];

const options = {
  compress: true,
  buffer_size: 0
};

const zxy = {z:5, x:5, y:12};

vtcomposite(tiles, zxy, options, function(err, result) {
  if (err) throw err;
  console.log(result); // tile buffer
});

Installation

Each make command is specified in Makefile

git clone [email protected]:mapbox/vtcomposite.git
cd vtcomposite

# Build binaries. This looks to see if there were changes in the C++ code. This does not reinstall deps.
make

# Run tests
make test

# Cleans your current builds and removes potential cache
make clean

# Cleans everything, including the things you download from the network in order to compile (ex: npm packages).
# This is useful if you want to nuke everything and start from scratch.
# For example, it's super useful for making sure everything works for Travis, production, someone else's machine, etc
make distclean

# This skel uses documentation.js to auto-generate API docs.
# If you'd like to generate docs for your code, you'll need to install documentation.js,
# and then add your subdirectory to the docs command in package.json
npm install -g documentation
npm run docs

Customizing the compiler toolchain

By default we use clang++ via mason. The reason we do this is:

• We want to run the latest and greatest compiler version, to catch the most bugs, provide the best developer experience, and trigger the most helpful warnings
• We use clang-format to format the code and each version of clang-format formats code slightly differently. To avoid friction around this (and ensure all devs format the code the same) we default to using the same version of clang++ via mason.
• We want to support LTO in the builds, which is difficult to do on linux unless you control the toolchain tightly.

The version of the clang++ binary (and related tools) is controlled by the mason-versions.ini, and uses mason-js uses to install the toolchain.

All that said, it is still absolutely possible and encouraged to compile your module with another compiler toolchain. In fact we hope that modules based on node-cpp-skel do this!

To customize the toolchain you can override the defaults by setting these environment variables: CXX, CC, LINK, AR, NM. For example to use g++-6 you could do:

export CXX="g++-6"
export CC="gcc-6"
export LINK="g++-6"
export AR="ar"
export NM="nm"
make

These environment variables will override the compiler toolchain defaults in make_global_settings in the binding.gyp.

Warnings as errors

By default the build errors on compiler warnings. To disable this do:

WERROR=false make

Sanitizers

You can run the sanitizers, to catch additional bugs, by doing:

make sanitize

The sanitizers are part of the compiler and are also run in a specific job on Travis.

Code coverage

To see code coverage you can view current results online at or you can build in a customized way and display coverage locally like:

make coverage

Note

Use // LCOV_EXCL_START and // LCOV_EXCL_STOP to ignore from codecov remotely. However, this won't ignore when running coverage locally.

For more details about what make coverage is doing under the hood see https://github.com/mapbox/cpp#code-coverage.

Benchmarks

Benchmarks can be run with the bench/bench.js script to test vtcomposite against common real-world fixtures (provided by mvt-fixtures) and to test vtcomposite against its predecessor compositing library node-mapnik. When making changes in a pull request, please provide the benchmarks from the master branch and the HEAD of your current branch. You can control the concurrency, iterations, and package of the benchmarks with the following command:

node bench/bench.js --iterations 1000 --concurrency 5 --package vtcomposite

And the output will show how many times the library was able to execute per second, per fixture:

1: single tile in/out ... 16667 runs/s (3ms)
2: two different tiles at the same zoom level, zero buffer ... 4167 runs/s (12ms)
3: two different tiles from different zoom levels (separated by one zoom), zero buffer ... 633 runs/s (79ms)
4: two different tiles from different zoom levels (separated by more than one zoom), zero buffer ... 1429 runs/s (35ms)
5: tiles completely made of points, overzooming, no properties ... 3846 runs/s (13ms)
6: tiles completely made of points, same zoom, no properties ... 50000 runs/s (1ms)
7: tiles completely made of points, overzoooming, lots of properties ... 3333 runs/s (15ms)
8: tiles completely made of points, same zoom, lots of properties ... 50000 runs/s (1ms)
9: buffer_size 128 - tiles completely made of points, same zoom, lots of properties ... 50000 runs/s (1ms)
10: tiles completely made of linestrings, overzooming and lots of properties ... 1163 runs/s (43ms)
11: tiles completely made of polygons, overzooming and lots of properties ... 254 runs/s (197ms)
12: tiles completely made of points and linestrings, overzooming and lots of properties ... 10000 runs/s (5ms)
13: returns compressed buffer - tiles completely made of points and linestrings, overzooming and lots of properties ... 5556 runs/s (9ms)
14: buffer_size 128 - tiles completely made of points and linestrings, overzooming and lots of properties ... 12500 runs/s (4ms)
15: tiles completely made of points and linestrings, overzooming (2x) and lots of properties ... 16667 runs/s (3ms)
16: tiles completely made of polygons, overzooming and lots of properties ... 1042 runs/s (48ms)
17: tiles completely made of polygons, overzooming (2x) and lots of properties ... 2174 runs/s (23ms)
18: return compressed buffer - tiles completely made of polygons, overzooming (2x) and lots of properties ... 2083 runs/s (24ms)
19: buffer_size 4096 - tiles completely made of polygons, overzooming (2x) and lots of properties ... 1087 runs/s (46ms)

Viz

The viz/ directory contains a small node application that is helpful for visual QA of vtcomposite results. It requests a single Mapbox street tile at z6 and uses the composite function to overzoom the tile at z7. In order to request tiles, you'll need a MapboxAccessToken environment variable and you'll need to run both a local tile server and a simple server for your viz application.

cd viz
npm install
MapboxAccessToken={token} node app.js
# localhost:3000

#in a separate terminal tab, run a simple server on a port of your choosing
#navigate to this port in your browser 
python -m SimpleHTTPServer x000

Tutorial

What is compositing?

Compositing is a tool to combine multiple vector tiles into a single tile. Compositing allows a user to:

• Merge tiles. Merges 2 or more tiles into a single tile at the same zoom level.
• Overzoom tiles. Displays data at a higher zoom level than that the tileset max zoom.
• Clip tiles. Clips the extraneous portion of a tile that’s been overzoomed.

Compositing: Merging 2+ Tiles

Let’s say you have two tiles at z5 - santacruz.mvt & losangeles.mvt. Each tile contains a single point that corresponds to one of the two cities. You could generate a single tile, santa_cruz_plus_la-5-5-12.mvt that contains both points by compositing the two tiles.

Source Tiles

santacruz.mvt - single point

losangeles.mvt - single point

Output Tile

Composited Tile: santa_cruz_plus_la-5-5-12.mvt

vtcomposite code:

const santaCruzBuffer = fs.readFileSync('/santacruz.mvt');
const losAngelesBuffer = fs.readFileSync('/losangeles.mvt');

const tiles = [
  {buffer: santaCruzBuffer, z:5, x:5, y:12},
  {buffer: losAngelesBuffer, z:5, x:5, y:12}
];

const zxy = {z:5, x:5, y:12};

composite(tiles, zxy, {}, (err, vtBuffer) => {
  fs.writeFileSync('/santa_cruz_plus_la-5-5-12.mvt', vtBuffer);
});

Compositing: Overzooming & Clipping Tiles

Let’s say we want to display our composited tile: santa_cruz_plus_la-5-5-12.mvt at z6.

We know that as zoom levels increase, each tile divides into four smaller tiles. We can calculate each the zxy of the z6 tiles using the formula outlined below. There are also libraries, such as mapbox/tilebelt that calculate the parent or children tiles for you, as well as other tile math calculations.

If the zxy is 5/5/12, the z6 children tiles are located at:

vtcomposite code:

const santaCruzAndLABuffer = fs.readFileSync('/santa_cruz_plus_la-5-5-12.mvt');

const tiles = [
  {buffer: santaCruzAndLABuffer, z:5, x:5, y:12}
];

//map request 
const zxy = {z:6, x:10, y:24};

composite(tiles, zxy, {}, (err, vtBuffer) => {
  fs.writeFileSync('/santa_cruz_plus_la-6-10-24.mvt', vtBuffer);
});

In this example, the tile being requested is at z6, but our source tile is a z5 tile. In this scenario, we must overzoom.

Each zoom level scales geometries by a power of 2. Thus, you can calculate coordinates at each zoom level knowing the original geometry and the (over)zoom factor.

  // original geometry = Santa Cruz tile coordinate at 5/5/12
  const originalGeometry = {x:637, y:1865};
  let x = originalGeometry.x;
  let y = originalGeometry.y;
  
  //increasing geometry size by a zoom factor of 1 
  const zoom_factor = 1; 
  
  const scale = Math.pow(2,zoom_factor); //1 << 1 
  
  //scale x and y geometries by the zoom_factor 
  let xScale = x*scale;
  let yScale = y*scale;
  
  //divide the scaled geometries by the tile extent (4096) to see the point moves to another tile 
  let xtileOffset = Math.floor(xScale/4096);
  let ytileOffset = Math.floor(yScale/4096);
  
  //subtract the difference between the x and y tileoffsets. 
  let xOffset = xScale - (xtileOffset * 4096);
  let yOffset = yScale - (ytileOffset * 4096);  

  //the xOffset and yOffset will be the x,y point at z6

Based off these equations, we know that resulting (x,y) point geometries for Santa Cruz and Los Angeles overzoomed at z6 are:

Santa Cruz point = [1274, 3730] at zxy 6/10/24
Los Angeles point = [90, 2318] at zxy 6/11/25

Clipping

Wait a second…! Los Angeles isn’t the tile we requested - {z:6, x:10, y:24} - it’s in {z:6, x:11, y:25}.

That means we need to clip the overzoomed geometries to only include the point(s) we need for tile {z:6, x:10, y:24}. Since Santa Cruz is the only geometry in {z:6, x:10, y:24}, we clip extraneous data, which means we remove any geometries that are not included in the z6 tile, but are included in the parent tile that’s been overzoomed - {z:5, x:5, y:12}. See ya Los Angeles!

Clipping with a buffer_size

In the example above, we clipped geometries based on the default tile boundaries (4096X4096). However, the composite function always us to have control over which geometries we include/exclude outside the requested tile when clipping. By passing in a buffer_size to the compositing function, we are able to explicitly state if we want to keep geometries outside the tile extent when overzooming.

Contributing and License

This project is based off the node-cpp-skel framework. Node-cpp-skel is licensed under CC0.

For more about vtcomposite contributing and licensing, see:

• vtcomposite license
• vtcomposite contribution docs