Comments (13)
It ought to be relatively easy to implement pmtiles merge a.pmtiles b.pmtiles
for case 1 above, throwing an error once any overlap in tiles is detected. Sounds like this would handle a good number of valid application.
from go-pmtiles.
I'm thinking we do this:
pmtiles merge INPUT_1.pmtiles INPUT_2.pmtiles INPUT3.pmtiles -o OUTPUT.pmtiles
- if the input is raster, it will fail if there is any overlap in addressed tiles
- if the input is vector, it will fail if there is any overlap in stated
vector_layers
in the metadata (we would need to trust the metadata to be correctly describing the tiles). Otherwise the merge will byte-concatenate the layers
We can maybe add a --overwrite-overlapping-tiles
option that will accept overlaps and choose the relevant tile from the first of the archives in order. I don't have any concrete plan to implement true tile-level merging.
minzoom/maxzoom issue
Finally there is an outstanding issue of how to deal with merging archives that do not have matching max zooms. If I merge one tileset that has max 14, and another that has max 15, my archive in total has a max of 15, but when I look at the 14 area in most browser and zoom into 15 I get nothing. This is a limitation of how maxzoom
is expressed in most clients like Leaflet, OL, MapLibre. I don't see a smart way to rectify other than also enforcing maxzoom
is uniform across all merged tilesets.
JSON metadata
We need to update the binary header like bounds, minzoom, maxzoom, # addressed tiles to be accurate after a merge. But if the JSON metatadata has keys like vector_layers
or tilestats
we may need to modify those. Since vector_layers
is a part of the specification we can figure out what the combined layer set is. But tilestats
may just need to be deleted, and any application-level JSON metadata could become incorrect after a merge operation.
cc @ZeLonewolf
from go-pmtiles.
There are a few details that complicate doing this correctly.
- If the if the tiles contained in each input pmtiles have no overlap, this is very easy to do
- If there is tile overlap, we can use the ordering of the inputs to determine priority:
input0.pmtiles
has priority. However, this may not work correctly for low zooms: imagine merging two raster tiles that cover Europe and South America. The Zoom 0 Tile is only correct if it contains the merging of the pixels. - If there is tile overlap for vector tiles, and the layers are disjoint, we can deserialize the overlapping tiles and combine their layers or even directly concatenate the protobufs.
- If there is tile overlap for vector tiles AND they have overlapping layer names, then we need to deserialize both tiles and then do a merging of their layers which requires rewriting the dictionary encoding of the keys/values.
In a perfect world we could accomplish all of the above with go-pmtiles, but it's preferable to first address the use cases people actually need (which do you need?). A major factor in determining these is whether it complicates the go-pmtiles dependencies. For example 2) above would require adding image processing libraries for PNG, JPG etc which will probably need a native dependency (we don't want any).
from go-pmtiles.
I have a Jekyll blog that I also use for travel blog posts. In the yaml meta data I can put a GEOJSON object. Then, a plugin loads currently mapbox with mapbox cloud data. I want to move the maps to selfhosting.
I do not want to have map data covering the entire world on high zoom levels, but instead only files for the area that I cover in my travel blog.
In my mapbox gl style file, I need to
(a) change for each geojson object the source to the corresponding pmtiles file. I do not know currently how I can pass through the file name from my javascript code to my style.json without inlining the entire style.json
(b) use only one pmtile file that is the concatenation of all pmtile files of the blog.
From what I read, (b) is so far not possible and difficult to implement. Hence, I would need to find a way how I can do (a) in the most elegant way.
from go-pmtiles.
@rriemann Is each region displayed on a separate page? If so, you can re-use only the layers
key of the style.json and then dynamically swap the sources
values in JavaScript to accomplish (a). Just changes the url
key of the source name to point at the pmtiles.
(b) will be difficult to implement as one big pmtiles if there are any low zoom tiles that overlap among the regions, since we'd have to do an individual tile merging for it to look correct.
from go-pmtiles.
In a perfect world we could accomplish all of the above with go-pmtiles, but it's preferable to first address the use cases people actually need (which do you need?).
Thank you for this detailed reply! We came across the following use cases recently
- Some datasets like the Copernicus DEM come in multiple files (e.g. 1° grid) and we can get one pmtiles file per grid-tile easily. But ideally we'd merge them at the end into a single DEM pmtiles file again. Adjacent and disjoint.
- We generate raster pmtiles for the same quadkey over multiple years. At the moment we create e.g. 2020.pmtiles 2021.pmtiles and so on. Ideally we'd merge them into a single pmtiles file where the years are represented as layers.
There are workarounds for now e.g. to merge the mbtiles files before converting to pmtiles but a pmtiles cat
would simplify things and provide a convenient way to handle use cases like the ones described above.
from go-pmtiles.
I don't know of any smart
MBTiles mergers that would accomplish the correct per-tile merging above - have any pointers?My suggestion for 1) is to build a single pmtiles from a VRT mosaic over the 1 degree grid cells. They won't be truly disjoint if there are any lower zoom levels generated, as those will almost certainly have some common tiles.
Use case 2 is 3) from my post above which is relatively easy to accomplish. The simplest way is to do direct PBF concatenation, relying on vector_layers
to enforce that there is no layer name overlap. If there is layer overlap the program should exit with an error. Then the algorithm is to iterate over tileIDs from archives A and B, taking into consideration deduplication/RLE (a little tricky), unzip both tiles, concatenate the protobufs (respect input ordering, probably) and re-zip the output tile
from go-pmtiles.
I would be interested in this feature as well, specifically when building raster tiles in the cloud it would be nice to be able to do something like "this .pmtiles
was from 0-11" and merge it with one that was from 12-18, etc. Given the cost of computing power, it would be useful to be able to not essentially waste-work if for some reason you had to try to recover from a particular point in the multi-day process of building the rasters.
from go-pmtiles.
I'd be interested in this feature, too - specifically case (3) described in #105 (comment)
I've used tile-join to similar effect, but based on felt/tippecanoe#10, I wonder if its support is more rudimentary than what's in go-pmtiles.
from go-pmtiles.
Thanks, this feature would help me for a few obscure development use cases when I want to demonstrate a change in OpenMapTiles and style it with Americana, the the examples I have are in different parts of the planet. For example, I'd currently like to render a sample that has Massachusetts, Yukon, and Spain to proof-out an OpenMapTiles concept (and hence I'm rendering it on openmaptiles-tools. In my admittedly one-off use case, I really don't care what happens in the case of tile collisions.
from go-pmtiles.
To address the issue of tilesets with uneven/variable max zoom level (e.g. protomaps/PMTiles/discussions/321), I see a use for merge + extrapolate(?). Assuming the original tileset has a maxzoom of 15-16:
- extrapolate z15 tiles into a full set of z16 tiles
- extract z16 tiles (partial coverage) from the original tileset - this is already supported
- merge results from 1 and 2 (with 2 taking priority)
This is not a common usecase but I feel it makes more sense to "fix" the data, instead of coming up with hacks for different renderers.
from go-pmtiles.
My use case:
I am working on a raster dem map that needs a detail of zoom 13 in one area but needs only zoom 10 in the other area.
If a render all the map in z13 is 100G, too much cause storage limitation. Solution:
-Create one area in z13 and the other in zoom 10 and then merge.
from go-pmtiles.
@bdon @daniel-j-h 👍 for pmtiles merge
it's preferable to first address the use cases people actually need (which do you need?)
Case 1 (no overlaps) would be an excellent start. Especially useful for larger datasets split into chunks.
Related: protomaps/PMTiles#403
from go-pmtiles.
Related Issues (20)
- s3-like compatibility HOT 8
- /health endpoint when serving HOT 10
- Add quiet mode HOT 4
- Slow query for totalTiles in MBTiles -> PMTiles conversion HOT 14
- convert Error parsing mbtiles HOT 4
- arm64 docker images HOT 3
- Corrupted file HOT 8
- SIGSEGV in `pmtiles serve` HOT 3
- dockerfile improvements
- Fails to convert world map due to /tmp space limitations HOT 2
- Invalid help comment for download command HOT 1
- change tmp file location HOT 2
- 404 when trying to serve local PMTiles using the pmtiles CLI HOT 3
- Add ability to get a subset of layers using pmtiles extract HOT 1
- Serving from public s3 bucket HOT 4
- Intergration as a library HOT 3
- CLI: `pmtiles convert INPUT.mbtiles OUTPUT.pmtiles` fails silently for MBTiles v 1.0 HOT 4
- Docker image cannot `convert` HOT 7
- server always sends 404 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-pmtiles.