protomaps / pmtiles Goto Github PK

When using pmtiles.js, network performance is a big bottleneck.

This is visible in the leaflet raster demo but mitigated by a very fast backend (read from a small pmtiles file), so response-time is ~15ms per tile.
Still you can see in the chrome dev tools that each fetch request is "stalled" (queued) until the previous one is answered.

My setup responds in ~150ms per tile, so time from scrolling to display is several seconds.

How can this be optimised?

• Reducing number of requests using Multipart Ranges does not seem possible without adding complexity to the leaflet integration.
• ... but according to chrome we're supposed to have up to 6 concurrent TCP connections per origin?

hi,
i'm using our own pmtiles tileset server, the test url is : https://tilesets.urbanease.io/cadastre/64/64102/without_protobuf.pmtiles
the location for seeing it is :bayonne, france latlng=[43.492949,-1.474841]

on the viewer, all seems to be ok bit leaflet preview don't work viewer
i'm using react and leaflet with the npm protomaps package version 1.19.0

my code is simple
const map = useMap(); const url = 'https://tilesets.urbanease.io/cadastre/64/64102/without_protobuf.pmtiles'; const layer = protomaps.leafletLayer({ url: url, id: 'cadastral', }); layer.addTo(map);

canvas are created but seems to be transparent, colors in paint_rules layer are good and opacity ok

i don't understand where is the pb bad pmtiles file? no compatibility with leaflet?
thx for help

I would like to propose some changes to the directory structure, but these might be totally irrelevant due to my misunderstanding.

Current directory entry is fixed at 17bytes, stores x,y as individual values, and requires x,y sorting order. I think all of those may benefit from a bit of restructuring:

• Combine x and y values into a single interleaved value (z-curve). This will make tiles more locale-clustered, possibly reducing the number of range requests (web).
• Make the combined x,y value size depend on the zoom level, rounding to the nearest byte boundary. zooms 0..3 -- 1 byte, 4..7 -- 2 bytes, etc. All leaf nodes are required to be the same zoom, making the directory entries the same size only inside a single directory block, rather than everywhere.
• For the root dir, it could still be 17bytes (although it seems 6 bytes is a bit too much -- zoom 16..19 only requires 5 bytes, so unless pmtiles wants to support more... and even that can be made flex)

I attempted to build my own tiles using OSM.

My workflow is:

osmium tags-filter socal-full.osm.pbf w/highway w/natural=water w/waterway=river w/water -o socal.osm.pbf
tilemaker --input socal.osm.pbf --output socal.mbtiles
pmtiles-convert socal.mbtiles socal.pmtile

TileMaker also layered in the coastline for me

When I view my map in MBTile format, I see all of the roads:

But viewing it in PMTiles after converting, I only see the ocean and lakes, no roads. I used default styling and tried to override them to see if I can force a render somehow.

integration tests across languages, reading/writing from all of:

• python
• C++
• java
• golang
• TypeScript

Am I correct in understanding that .pmtiles are derived from .osmx databases?

If that is the case is there public code/tooling to do the conversion? I've been poking around the various repos and documentation but can't seem to find it.

maplibregl should be maplibre_instance on this line.

This should allow O(log n) lookup of tile membership in a directory without creating a copy.

Unsure why this is but I can no longer use this server:
pmtiles-serve --cors-allow-all --bind 0.0.0.0 wrecks.pmtiles 9999

Only see some 400s in the terminal:

I'm trying to convert a 66 GB file and it quickly consumes 32GB ram then stops.

Is there a solution to converting large files besides launching a super huge VM to handle conversions?

Using master.
Here is a random mbtiles before/after the following roundtrip

PYTHONPATH=$PWD bin/pmtiles-convert my.mbtiles foo.pmtiles 
PYTHONPATH=$PWD bin/pmtiles-convert foo.{pmtiles,mbtiles}

• left: my.mbtiles
• right: foo.mbtiles has "horizontal bands" shuffled
  

I suspect something like #8

The pmtiles writer ungzips tiles before adding to the archive. Would it be possible to have an option to keep them gzipped? It reduces the file size substantially, and clients could use something like pako to inflate on the client side until transfer-encoding: gzip is better supported.

From a quick look at the history it seems this error may have snuck in to version 2.0.0

I swapped out lines 88-89 for lines 64-67 and it worked as expected :)

This is the default behavior for Tippecanoe.

• SVG tile previews should be zoomable
• Should be able to drill down into leaf directories
• should be able to preview vector tiles in leaflet
• SVG should be feature-level inspectable
• map preview should be feature-level inspectable
• see directory Len and header-level metadata
• inspect SVG with mismatched extents
• correctly read tile and map hash states
• split out "individual tile inspector" to separate page
• view raw zxy and TIleJSON endpoints

Some common TileMatrixSet definitions (Appendix D http://docs.opengeospatial.org/is/17-083r2/17-083r2.html) do not fit in pyramids because ID 0 has more than one tile. We can define a uint8 TileMatrixIDOffset that translates between zoom levels and Tile IDs, allowing tiles like Z=0 X=1 Y=0 to be addressable by internally shifting it to Z=1 in a way that is handled by the PMTiles implementation.

(edited as summary) The current Python pmtiles-convert program deduplicates by hashing each using the hash builtin: https://docs.python.org/3/library/functions.html#hash which produces a 64-bit value on 64-bit systems.

A typical basemap use case has about 40 million unique tile contents for zooms 0-14; zoom 15 has 100-200 million unique tile contents.

See this on Hash Collision Probabilities for formulas.

A collision between two hash values is unacceptable because that means the PMTiles archive will replace the collided tile with the wrong contents.

The approximate "birthday paradox" probability of a collision giving a (perfectly distributed) hash function of possible values n where there are k values being hashed is: k^2 / 2n.

• For 64-bit hashes and 40 million unique tiles, the probability of a collision is ~1 in 23,000
• For 64-bit hashes and 200 million unique tiles, the probability is ~1 in 922

The probability of collisions in these cases is unacceptably high for the general use case of 100m+ tiles. A general-purpose deduplication implementation should use 128 bit hashes.

• For 128-bit hashes and 40 million unique tiles, the probability of a collision is 1 in 4.25 * 10^21
• For 128-bit hashes and 200 million unique tiles, the probability is 1.70 * 10^20

Hello,
how can I create my own pmtiles?

It seems like go-pmtiles is set up to read from cloud storage (S3, Google Storage etc), but the python version here is not. Is that correct? How hard would it be to add that? I use Google, but an S3 example would be great.

If the Reader API is robust, I can add it into a flask/fastapi endpoint pretty trivially, implement my own authentication etc, but it definitely seems like being able to read the tiles from a file in Storage (rather than locally) is the really powerful use case here.

Currently the loading of the root directory is in the constructor.

This is not desirable when the layer is not the default one, eg when switching between several layers with leaflet Layers Controls, the useless queries may be costly on backend side.

I guess we can start with PMTiles().root ===undefined and load it on first access (at the cost of initial load time)?

(There are also leaflet layer control event if needed)

Some hosts (like github pages) have maximum file sizes. Alternatives like https://github.com/phiresky/sql.js-httpvfs provide a way to split the tile archive until it is less than that max file size (https://github.com/phiresky/sql.js-httpvfs/blob/master/create_db.sh). Would it be possible for the pmtiles reader and writer to optionally support splitting a pmtiles file?

Problem: Now pmtiles-serve directly uses y value in sqlite table tiles from MBTiles, which is TMS scheme.

In fact, MBTiles always follow TMS scheme. So if we want to serve tiles with XYZ scheme, the y value should be

# y is the value in MBTiles
2^z - 1 - y 

As it's missing the tpe_sample.pmtiles file

Hi, thanks for the awesome project!

I'm trying to use pmtiles as basemap in my map project, and I would like to do some load testing on it. Normally I use siege for this kind of testing and I'll feed URLs with randomised zoom level and x, y values.

I wonder if I could do something similar for pmtiles? If so, how would the URL look like in this case?

FYI, I'm using MaplibreGL and I inserted PMTiles in my code like this:

"tiles": ["pmtiles://sample.pmtiles/{z}/{x}/{y}"]

So I'm not sure what would be a URL in this case.

Thank you.

Options:

• Gzip compression - requires a library like pako, may be expensive
• Cap'n proto packing: https://capnproto.org/encoding.html
• Protobuf varints: https://developers.google.com/protocol-buffers/docs/encoding

Use cases:

• Dense tile pyramids
• Sparse pyramids (tippecanoe output)

The implementation is much simpler at the expense of discontinuities.

Related to Tippecanoe

Before I start poking around in earnest I wanted to see whether this error was a "known-known" or otherwise something that you'd recognized faster than me.

The error is triggered by running a pmtiles-convert on a MBTiles database derived from the Geofabrik downloads, using tilemaker.

$> wget https://download.geofabrik.de/north-america/us/california/norcal-latest.osm.pbf

$> tilemaker --input norcal-latest.osm.pbf --output norcal-latest.mbtiles

$> pmtiles-convert norcal-latest.mbtiles norcal-latest.pmtiles
Traceback (most recent call last):
  File "/usr/local/bin/pmtiles-convert", line 49, in <module>
    result = writer.finalize(metadata)
  File "/usr/local/lib/python3.9/site-packages/pmtiles/writer.py", line 98, in finalize
    root = [(group[0],list(group[1])) for group in itertools.groupby(self.tiles,key=by_parent) if group[0][0] == 0][0]
IndexError: list index out of range

I just realized the version from the README:
https://unpkg.com/[email protected]/pmtiles.js
is quite outdated compared with the version in the repo
https://github.com/protomaps/PMTiles/blob/master/js/index.src.mjs

For example it does not pass "options" to leaflet ie return new cls() instead of return new cls(options).

Naively I tried https://unpkg.com/pmtiles@latest/pmtiles.js but it points to 0.3.1 which has different files so should be https://unpkg.com/browse/pmtiles@latest/index.js I guess?

https://github.com/protomaps/PMTiles/blob/master/python/pmtiles/reader.py#L61

Hi! I'm new to this project. I tried running some html files in the example section in Safari/Chrome but maps were not shown. I got the following errors:

I couldn't figure out why fetch is aborted. Is there something I'm missing here?

Thank you.

If you run pmtiles-convert --gzip WFIGS_-_2022_Wildland_Fire_Perimeters_to_Date.mbtiles WFIGS_-_2022_Wildland_Fire_Perimeters_to_Date.pmtiles (MBTiles file is attached), it fails with the following error:

(...)
  File "(...).venv/lib/python3.9/site-packages/pmtiles/writer.py", line 131, in _write_header
    assert len(metadata_serialized) < 140625
AssertionError

metadata_serialized len is 168067.

$ pip show pmtiles
Name: pmtiles
Version: 2.0.0

WFIGS_-_2022_Wildland_Fire_Perimeters_to_Date.mbtiles.zip

Right now, byte serving is confirmed by issuing a HEAD request for the first 512k bytes and checking the response is 206. However, many CDNs such as GitHub pages, Azure storage and AWS Cloudfront return 200 instead of 206; allowing this to go through is controlled by the allow_200 option. This is confusing and places too much burden on the developer.

We can instead skip the preflight HEAD check completely, do a GET for the first 512k, ignore the status code and check the Content-Length is 512000 (Content-Length is a "safe" header that will be exposed to scripts: https://developer.mozilla.org/en-US/docs/Glossary/CORS-safelisted_response_header ). If it is not 512000, we can use AbortController to abort the request and reject the promise, making PMTiles fail in an obvious way.

I'm using Cordova/Ionic in order to serve offline tiles.
I'm doing this by leveraging sqlite-ext to open a mbtiles file on the device and serve the tiles from there.
Is there a way to to use pmtiles in Cordova in similar manor?
From the documentation it seems that there's a need to use memory mapped file or something similar, have you experimented with this?

• moving away from storing ArrayBuffers
• get_bytes API should be accompanied by a get_directory API that returns List/Array objects in the native language

This is a gotcha when working with overzooming

see https://maplibre.org/maplibre-gl-js-docs/style-spec/sources/#vector-maxzoom

The current "proof of concept" JS output isn't quite production ready; I'm going to add a few things

• Change pmtiles.js into a pmtiles.mjs source file
• add a build/transform step (I prefer esbuild, is there any strong reason we need to bring in something like Babel for a very simple library?)
• Output is an IIFE file that is not versioned but is on NPM

also:

• CORS headers
• bind to other address
• reminder that python server is not for production use

If a dataset is updated on object storage, and a client has a directory cached, file offset/lengths will be invalid and the PMTiles client will fetch garbage for tiles.

We can detect this situation via ETag:

1. do all storage platforms allow this tag to be exposed via CORS?
2. how many retries should we allow? (assuming datasets change infrequently)