mapbox / mapnik-vector-tile Goto Github PK

It appears that with high tolerance settings the start and end vertex may be discarded when encoding lines. We need tests to ensure this cannot occur and fix any bugs around tolerance handling to prevent it.

Should also take a look at #29 at the same time.

Tasks include:

• - ensure that any internal move_to commands are preserved for multilines that happen to be encoded this way (b515cf7).
• - ensure that move_to commands are never dropped (8950587). For example if two move_to commands are found in sequence we currently drop the second if the tolerance is such that it is not a valid movement. But this could create gaps - the last move_to in a sequence is the critical one to keep.
• - ensure that the last vertex in lines is never dropped no matter how high the tolerance (2664718)
• - fix tilelive-bridge tests after 2664718: https://gist.github.com/springmeyer/f04660a83823a03121cd
• - should we try to drop all but the last if multiple move_to are found in sequence?
• - ensure that the last vertex in polygons is never dropped no matter how high the tolerance
• - ensure that the line_to vertex before an internal move_to in lines is never dropped no matter how high the tolerance
• - ensure that the line_to vertex before an internal move_to in polygons is never dropped no matter how high the tolerance

Just needs a quick npm publish.

It is listed as uint64 but in fact feature ids can be negative in mapnik (to support OSM negative ids) and the type is long long (mapnik::value_integer). We either need to change the proto or (more ideally) implement zigzag encoding at both tile encoding and decoding.

This will be a breaking change to the tile format.

@springmeyer, as we just discussed on chat, I have been trying to use multipoint geometries in Mapbox Studio.

As a test, I uploaded this GeoJSON file to mapbox.com as map ID enf.tqgmbo6r.

It looks like each MultiPoint draws as a single point at the centroid of the points within each feature within each tile rather than drawing all the points, regardless of how marker-multi-policy is set:

As individual points (enf.6vjqncdi) it looks like this:

but the file is twice as large with each point in its own feature.

Hello,

I'm trying to render vector tiles using node-mapnik mapnik.VectorTile feature:

var map = new mapnik.Map(256, 256);
var mercator = new sphericalmercator({size: 256});
map.loadSync("../openstreetmap-carto/openstreetmap-carto.xml");
var bbox = mercator.bbox(0,0,0, false, "900913");
map.zoomToBox(bbox);
var vtile = new mapnik.VectorTile(0, 0, 0);
map.render(vtile,{},function(err,vtile) {fs.writeFileSync('0.vector.pbf', vtile.getData());});

I use the standard openstreetmap-carto stylesheets. When I render it to png tiles, the 0-level zoom tile is a few kilobyte file that contains a simple image of the earth. When I render it to 0-level zoom vector.pbf using the code, the result .pbf file is a large file that seems to have all the map features on the 0-level zoom. How can we use the same zoon-related rules to keep 0-level zoom as simple as it rendered on png? Thank you.

again and again :)

Jérémy.

https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_processor.hpp needs the same treatment as mapnik saw in mapnik/mapnik@aaae8b1#diff-e6b967728b887a3988f89dd623d07554

We have several fields in the proto that are marked optional but are expected to exist as far as how the code implementation works:

• feature id
• layer extent
• feature geomType.

feature id in particular does not have a default.

We need to either change the proto file or adapt the code.

When using toGeoJSON, it can be useful to set buffer-size = 0.
But https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_processor.hpp#L132
ignores it because of the way boost::optional works.

I'd try to propose a patch if i was sure it's the right way to fix this.
Is it ?

vtile.isSolid for raster layers always returns false, which is correct but accidental. Needs a fresh look on a rainy day.

Currently 'features = 2' are encoded in advance to 'keys = 3' and 'values = 4' [1]. When processing tile data as stream this means that feature messages cannot be processed further until keys and values are read. If features would follow keys/values one could build complete tags as soon as a feature is read and pass it down the pipeline.
Please consider for the next protocol revision to use 'features = 6' and mark the default encoding order as required.

[1]https://developers.google.com/protocol-buffers/docs/encoding#order

Some features end up having bogus feature IDs that are numbers like 18446744073449012372. Is that a result from uninitialized memory?

Currently you can't send a tile to the tileinfo util over stdin. This would be useful for inspecting tiles on remote servers

Currently we use the protobuf format and the libprotobuf c++ bindings for encoding and decoding vector tiles. But benchmarking has shown with the standard libprotobuf API that memory allocation of stl types (mostly std::string) is a bottleneck for parsing messages, even with the help of tcmalloc/jemalloc. One strategy to mitigate this is object re-use but this approach does not mesh well with our target usage from node.js.

Ideas:

• Recently capnproto was released by @kentonv, and the C++ documentation captures perfectly what we have encountered and more: capnproto/capnproto@64286a5
• We could write our own lazy protobuf parser to avoid incurring the overhead of memory allocation for the temporary protobuf object. The benefit of this could be huge when only some of the layers in the vector tile are needed for rendering (only those used in a given style), or some of the features geometries are requested for rendering based on rules
• Potentially leverage https://github.com/haberman/upb/, although it appears to have been pulled into internal google development and the last commit broke the public code.
• Look into using set_allocated_foo(Type* foo)

I'm not 100% sure if gdal 1.11.1 is the cause here, but on Fedora 20 with gdal 1.10.1 the tests pass fine but on Fedora Rawhide with gdal 1.11.1 I am getting a failure:

./test/run-raster-test
ERROR 1: Only OGC WKT Projections supported for writing to GeoTIFF.
��Z� not supported.


run-raster-test is a Catch v1.0 b26 host application.
Run with -? for options

-------------------------------------------------------------------------------
vector tile output 1
-------------------------------------------------------------------------------
test/raster_tile.cpp:21
...............................................................................

test/raster_tile.cpp:52: FAILED:
  CHECK( 1 == tile.layers_size() )
with expansion:
  1 == 0

The initial error about the projections seems to be a gdal error.

otherwise we will hit:

In file included from ../src/mapnik_vector_tile.cpp:19:
../node_modules/mapnik-vector-tile/src/vector_tile_util.hpp:25:25: error: no type named 'Polygon' in namespace 'ClipperLib'
            ClipperLib::Polygon clip_box;
            ~~~~~~~~~~~~^
../node_modules/mapnik-vector-tile/src/vector_tile_util.hpp:42:29: error: no type named 'Polygons' in namespace 'ClipperLib'
                ClipperLib::Polygons geometry;
                ~~~~~~~~~~~~^
../node_modules/mapnik-vector-tile/src/vector_tile_util.hpp:43:29: error: no type named 'Polygon' in namespace 'ClipperLib'
                ClipperLib::Polygon polygon;
                ~~~~~~~~~~~~^
../node_modules/mapnik-vector-tile/src/vector_tile_util.hpp:91:29: error: no type named 'Polygons' in namespace 'ClipperLib'
                ClipperLib::Polygons solution;
                ~~~~~~~~~~~~^
../node_modules/mapnik-vector-tile/src/vector_tile_util.hpp:94:25: error: no member named 'AddPolygons' in 'ClipperLib::Clipper'
                clipper.AddPolygons(geometry, ClipperLib::ptSubject);
                ~~~~~~~ ^
../node_modules/mapnik-vector-tile/src/vector_tile_util.hpp:95:25: error: no member named 'AddPolygon' in 'ClipperLib::Clipper'
                clipper.AddPolygon(clip_box, ClipperLib::ptClip);
                ~~~~~~~ ^

While investigating shape triangulation for Mapbox GL JS, I discovered a really weird bug: some shapes in VT that are supposed to have one ring of vertices actually return several duplicated rings. I mean, instead of [ring] it has [ring, ring, ring] in the data, where ring is an array of coordinates.

Not sure if it's a Mapnik VT issue, data issue, vector-tile-js or anything else. Any ideas will be appreciated.

One particular example in http://b.tiles.mapbox.com/v4/mapbox.mapbox-streets-v6-dev/15/9370/12535.vector.pbf

The green park around the Embassy of Saudi Arabia in DC is returning 3 rings instead of one.

cc @yhahn @springmeyer

Compositing vtiles requires a decision on how to handle layer names within and across vtiles that might clash or be duplicates. The current plan is to allow duplicate names / render everything, but allow calling applications to differentiate (say for styling) the features that come from mapnik::vector::tile_datasource.

So, two vtiles composted together that both have a buildings layer but come from difference tile sources could be differentiated like:

#buildings {
  [source='data-virginia'] {
     polygon-fill:red;
  }
  [source='data-DC'] {
     polygon-fill:blue;
  }
}

Where source or some other magic attribute keyword would report the name of the vtile as originally passed to the compositing code:

var sources = [{name:'data-virginia',source:vtA},{name:'data-DC',source:vtB}

Currently we support dropping coincident vertices within a geometry, but not across features. In the case of many dense point features we could provide a mechanism to drop entire features whose point geometry are fully coincident with previously rendered points. This would not be ideal for cases where a client might want to keep all features interactive/clickable, but for all other cases this would drastically reduce the payload of the tiles.

Open issues if implemented:

• Which points should "win": the first rendered or the last?
• Optimized methods for checking for duplication in pixel space across features - quadtree/kdtree/rtree?

tolerance seems to usually be set to 32, and I somewhat understand this to make sense in that we want to ignore drawing a stroke twice on something that is within the same pixel. 32 makes sense to me, because that would be what you need to get a valid pixel that is hypoteneuse from a given pixel. Then, to do stroke for adjacent pixels, sharp_turn_ahead would flag as true and the stroke would be made as an override.

In particular, I'm looking at:

This means that tolerance is only effective for obtuse angles, and acute / 90deg angles would continue to be super sampled x 16. Why do we want to do this at all? My guess is that when mapnik renders the pbf to png, it will anti-alias sharp turns and points that exist within the previous pixel. Is this why?

I do see that turning off tolerance altogether for the most precise pbf zoom level makes sense, because we want non-simplified geometry for all subsequent rendered tiles. I suppose not using a path_multiplier would take away this benefit, Still, we could potentially just use the path_multiplier of 16 for the closest zoom level and not use it for others.

Need to look into this: If polygon geometries are not clipped then the cmd ends up being 7 at https://github.com/mapbox/mapnik-vector-tile/blob/master/src/vector_tile_backend_pbf.hpp#L227

This will enable removing the last hardcoded 256 value in vector_tile_backend_pbf.hpp

The tolerance threshold is designed to optimize encoding by throwing out no-op moves and not to simplify shapes. We should try harder to keep critical shapes, like sharp/right angles. One way to do this would be to look ahead one vertex and if the angle formed by the next move is > something like 30 degrees then we could avoid skipping the vertex that forms that angle. Related to #39.

Running list of ideas for more space optimizations:

• - Make the first moveto command implicit (i.e. 1 moveto command comes at the beginning of every geometry). In the case of points, we'd just encode the two points.
• - Avoid explicitly storing the geometry type, instead detect points (2 commands) vs lines (>= 2 commands, not closed) vs polygons (>= 3 commands, closed).
• - Sort key/value dictionary by frequency
• - Eliminate repeated close path commands in the case of geometry transformations leaving them behind
• - Use shortest encoding for integers (negative/unsigned ints)
• - Enable way to use float encoding, rather than double
• - Expose option to drop encoding feature id's if not needed/meaningful

Hi,
it would be very nice of you to push the git tag for version 0.5.2.

Thank you !
Jérémy.

We could build up an rtree when the tiles are parsed, or we could simply do a poor mans bbox check to throw out features at parse time that do not intersect with the query bbox.

We will need to be careful not to throw out features within the query buffer to avoid cut off labels that might need rendered on adjacent tiles.

The top-level README file refers to "vector-tile" specifications, but the C++ tileinfo code seems to only be able to read "OpenStreetMap PBF" file format (this), although its README says it can give you information about "vector tiles".

Are they just sharing the filename extension ? Would it make sense to define a different one for the newer format ?

See also mapbox/vector-tile-spec#20

We could attempt to detect straight lines formed by >= 3 points and throw out the internal point while ensuring the outside points are kept.

Hi:

I am not sure if this is the right place to post my question, since I though the Mapnik is a library for rendering map, that's to say Mapnik will generate map image according to a map data source(shapfile and etc) and a Mapnik style.

So I think the most important feature of Mapnik is its rendering feature.

It seems that it have nothing to do with generating vector tiles? Isn't it?

Do I miss anything?

Currently, as encoded, layers are pretty simple and dumb. This was an intentional design but has limitations. It makes it hard to know critical details about them without fully decoding them or without having metadata from elsewhere.

For example:

• Tiles with differing z,x,y should be able to be concatenated. If we encoded z,x,y inside the layer then decoding could be easily sensitive to the individual layer's tile extents.
• If we encoded the buffer-size used to originally clip vtile geometries then we could easily know this extent upon decoding without decoding and introspecting each geometry.

Seeing that $PATH is not being respected as I would assume: a different protoc is being used at /usr/bin/protoc instead of a custom protoc what is preferred in the parent env. Need to look into what npm is doing: perhaps it is dropping the PATH (or node-gyp)

I'm trying to find more information on how to work with terrain data in vector tile format. There is Working with terrain data at mapbox.com which details how to render maps in TileMill, but it uses SRTM data converted to raster data (GeoTiff).

Recently mapbox-gl-native was released which uses terrain data in vector tiles.

I'd like some guidance on how to add terrain data, probably from CGIAR or another source, into postgis and later generate and style vector tiles in mapnik.
I'm particularly interested in hillshades and slopes, since contours can be imported with gdal_contour.

Any information would be appreciated.

Need test coverage for potential error conditions which might lead to incomplete or corrupt vector tiles.

Clipping and simplification of polygon geometries often makes the polygons non-simple (with intersecting & touching segments). This is hard to handle when trying to tesselate such polygons for rendering (both on native & JS). Native uses Clipper (specifically SimplifyPolygon with StrictlySimple) to fix the geometries. It has a port for JS, but applying it there would take a lot of precious CPU time, so it would be great to fix those problems on the VT side, using Clipper there.

Intersecting edges after simplification:

Degenerate edges after clipping:

cc @springmeyer @yhahn

Currently as far as bindings, we have fully fledged node.js bindings, but for python currently only a set of examples for usage exists at https://github.com/mapbox/mapnik-vector-tile/tree/master/python which do not actually use mapnik.

Task here is to write python bindings against mapnik to enable things like a custom provider in TileStache.

I had an error while trying to install TM2; It failed while doing this:

protoc -Iproto/ --cpp_out=./src/ ./proto/vector_tile.proto

the full error is on this gist https://gist.github.com/ivancamilov/5746598. Thanks in advance for your help!

Do geometry rings have a defined winding order in vector tiles? E.g. counter-clockwise for outer rings and clockwise for rings? Or should I expect any order of any particular ring?

Winding order isn't mentioned anywhere in spec, but some polygon algorithms depend on a certain winding order. Also, some algorithms (especially triangulation) need to know if a polygon ring is a hole or an outer ring, which could also be derived from winding order if geometry is flattened.

@artemp is seeing: https://gist.github.com/artemp/5d3cd1514c7b03be76a3

Looks like slightly different raster sizes perhaps due to different tiff/jpeg/png versions.

Likely the best solution is to start comparing images using a pixels and a slight tolerance rather that strict encoded image sizes.

Mapnik encodes each part of a multipart polygon in a separate array. This is to enable styling control over marker placement and labeling: you might wish for a placement per polygon part or a on the largest polygon part.

As far as Mapnik rendering is concerned (AGG renderer) and SVG rendering models in general, geometry parts can be stored and rendered one at a time or in a batch (flattened into one path).

Currently the vector tile encoding represents the flattened path, which was a shortcut taken to simplify the encoding.

So when encoding (creating) a vector tile we loop over all possible geometry parts and then they each get added to the single geometry array of a protobuf feature. (and incidentally the last geometry type wins in the case of a geometry collection meaning that collections are not supported).

This has one limitation: when decoding geometries we cannot easily distinguish the original geometry parts.

Currently this means that we take the flattened geometry as is. In short we assign one geometry to one feature even if the original data included multiple geometries. This means that when later decoded back into mapnik objects and styled you'll only get one marker for multigeoms rather than for each geometry (the default when geometries are stored properly). The default text labeling in mapnik is to only label the largest geometry (relevant for polygons and lines) so this bug will be less noticeable but still likely result in different labeling on a vtile vs the original data (until this is fixed).

One decoding fix is to split each geometry into parts for each move_to encountered. This means that move_tos which originally represented independent geometry parts (true multipolygons) will be handled correctly but move_tos representing inner rings in polygons with holes will be incorrectly split.

It seems like the vector::processor sometimes fails to get some lines (from a shapefile) on the buffered part of the tile, especially when the line seems to be north/south of the tile.

I have rendered all geometries in some tiles and the resulting image shows that some smaller roads and (which are more problematic for my case) label-lines are removed in the buffered part (compared to the surrounding tiles actual data/image). This results in cut off labels (the line is the center of the text) for some tiles.

I have tried to take a larger buffer but it doesn´t fix it. Is this a known/unknown problem or have I misunderstood something?

Hey @springmeyer ,
Ran into a couple shapefiles today that are not rendering in tm2. The metadata coming out of the shapefile unpacker looks good, so I'd like to pick your brain about what Mapnik is doing.

I have a shapefile of Chicago Parks with empty tiles, except for 0/0/0.pbf (tm2 displays nothing, but the tile's json shows data). Odd because the shapefile unpacker is recognizing other data and setting max zoom to 12, which seems accurate for a dataset of Chicago Parks.

Chicago Parks dataset bounds:

-87.84150392833166,
41.65312799225436,
-87.52057860367273,
42.0219983177277
]

Is mapnik possibly refusing to draw the data at z0? Has this ever happened for any other datasets in the past?

Another possible factor is I'm using the Pyramid Scheme to optimize performance. Could mapnik be thinking this z0 tile is empty and skipping over it? How does mapnik decide what to translate as "empty" when using the Pyramid scheme?

Thanks for any input!

Hi,

I read about vector tiles in MapBox blog and found it interesting. I am trying to understand the features built in this project.

Is there an example python program which renders bitmaps from protobuf messages or vector tiles?

The tests pass fine on 64 bit systems, but on a 32 bit x86 system I get a failure:

test/vector_tile.cpp:39: 4096 == f.geometry(1) failed for: 4096 == 4094

The tests are failing to build in the 0.5.5 code:

In file included from ./src/vector_tile.pb.h:25:0,
                 from ./src/vector_tile_backend_pbf.hpp:12,
                 from test/vector_tile.cpp:21:
/usr/include/google/protobuf/extension_set.h:183:21: error: reference to 'vector' is ambiguous
                     vector<const FieldDescriptor*>* output) const;
                     ^
In file included from /usr/include/c++/4.8.3/vector:64:0,
                 from test/catch.hpp:355,
                 from test/vector_tile.cpp:3:
/usr/include/c++/4.8.3/bits/stl_vector.h:210:11: note: candidates are: template<class _Tp, class _Alloc> class std::vector
     class vector : protected _Vector_base<_Tp, _Alloc>
           ^
In file included from test/vector_tile.cpp:13:0:
./src/vector_tile_projection.hpp:12:37: note:                 namespace mapnik::vector { }
 namespace mapnik { namespace vector {
                                     ^
In file included from ./src/vector_tile.pb.h:25:0,
                 from ./src/vector_tile_backend_pbf.hpp:12,
                 from test/vector_tile.cpp:21:
/usr/include/google/protobuf/extension_set.h:183:21: error: 'vector' has not been declared
                     vector<const FieldDescriptor*>* output) const;
                     ^
/usr/include/google/protobuf/extension_set.h:183:27: error: expected ',' or '...' before '<' token
                     vector<const FieldDescriptor*>* output) const;
                           ^
In file included from ./src/vector_tile.pb.h:25:0,
                 from ./src/vector_tile_backend_pbf.hpp:12,
                 from test/raster_tile.cpp:12:
/usr/include/google/protobuf/extension_set.h:183:21: error: reference to 'vector' is ambiguous
                     vector<const FieldDescriptor*>* output) const;
                     ^
In file included from /usr/include/c++/4.8.3/vector:64:0,
                 from test/catch.hpp:355,
                 from test/raster_tile.cpp:3:
/usr/include/c++/4.8.3/bits/stl_vector.h:210:11: note: candidates are: template<class _Tp, class _Alloc> class std::vector
     class vector : protected _Vector_base<_Tp, _Alloc>
           ^
In file included from test/raster_tile.cpp:8:0:
./src/vector_tile_projection.hpp:12:37: note:                 namespace mapnik::vector { }
 namespace mapnik { namespace vector {
                                     ^
In file included from ./src/vector_tile.pb.h:25:0,
                 from ./src/vector_tile_backend_pbf.hpp:12,
                 from test/raster_tile.cpp:12:
/usr/include/google/protobuf/extension_set.h:183:21: error: 'vector' has not been declared
                     vector<const FieldDescriptor*>* output) const;
                     ^
/usr/include/google/protobuf/extension_set.h:183:27: error: expected ',' or '...' before '<' token
                     vector<const FieldDescriptor*>* output) const;
                           ^

Compiler is gcc 4.8.3 and mapnik is 2.2.

I think the README.md file and especially the section describing installation of mapnik-vector-tile for Ubuntu lacks the dependency toward "libspatialindex-dev"

I had to install it to be able to install 'rtree' with 'pip'

Regards,

Thibault D.

Hi,

I am currently working from the .proto file to understand the Mapnik Vector Tile's functionment. I have a question about the test file, when you create the point called null island at (0, 0) in the Spherical Mercator projection.

I thought the coordinates of the tiles produced would be between 0 and extent - 1 = tile_size * path_multiplier - 1 = 4095 ( https://bit.ly/1i27XxI ), so for the single tile at z=0, the point is exactly in the middle of the tile: in our case at (2047, 2047). However, you check that the coordinates are 4096: https://bit.ly/1cODTG0 . I couldn't find the reason of this factor 2, do I miss something?

This change needs to be partially rolled back (39f907f#diff-406a74d25616e1fcc5cc419024f6cd8f).

The fix for closing polygons will stay in place and is essential for node-mapnik vtile.query hit_test results which are used by carmen.

The fix to return multipaths as multiple mapnik::geometry_type, while fixing display of labels and markers on multipart geometries, is too expensive for memory until point and small geometry storage is improved in mapnik - mapnik/mapnik#2151.