protomaps / go-pmtiles

Single-file executable tool for working with PMTiles archives

go-pmtiles

The single-file utility for creating and working with PMTiles archives.

Installation

See Releases for your OS and architecture.

Creating a PMTiles archive from MBTiles

Convert an MBTiles archive:

pmtiles convert INPUT.mbtiles OUTPUT.pmtiles

Create a PMTiles archive from a larger one

pmtiles extract INPUT.pmtiles OUTPUT.pmtiles --region=REGION.geojson
pmtiles extract https://example.com/INPUT.pmtiles OUTPUT.pmtiles --region=REGION.geojson
pmtiles extract INPUT.pmtiles OUTPUT.pmtiles --maxzoom=MAXZOOM --bucket=s3://BUCKET_NAME

• --region a GeoJSON Polygon, Multipolygon, Feature, or FeatureCollection
• --maxzoom=13, --minzoom=12 extract only a subset of zoom levels, see docs for details
• --download-threads parallel requests to speed up downloads
• --overfetch extra data to download to batch small requests: 0.05 is 5%

Uploading

Upload an archive to S3-compatible cloud storage:

# requires environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY credentials
pmtiles upload INPUT.pmtiles REMOTE.pmtiles --bucket=s3://BUCKET_NAME

Inspecting archives

pmtiles show INPUT.pmtiles
pmtiles show INPUT.pmtiles --bucket=s3://BUCKET_NAME

Serving Z/X/Y tiles

This section covers running a Z/X/Y tile server proxy for clients that read only those URLs. The simplest way to consume PMTiles on the web is directly in the browser: see the JavaScript examples.

Serve a directory of archives from local or cloud storage as a Z/X/Y endpoint:

pmtiles serve .
# serves at http://localhost:8077/FILENAME/{z}/{x}/{y}.mvt

pmtiles serve / --bucket=s3://BUCKET_NAME
pmtiles serve prefix --bucket=s3://BUCKET_NAME

For production usage, it's recommended to run behind a reverse proxy like Nginx or Caddy to manage HTTP headers (Access-Control-Allow-Origin, Cache-Control...) and SSL certificates.

pmtiles serve [FLAGS] BUCKET

• --cors=ORIGIN set the value of the Access-Control-Allow-Origin. * is a valid value but must be escaped in your shell. Appropriate for development use.
• --cache-size=SIZE_MB set the total size of the header and directory LRU cache. Default is 64 MB.
• --port=PORT specify the HTTP port.

Metadata is served at the URL path /<archive_name>/metadata.

Tiles are served at the URL path /<archive_name>/<z>/<x>/<y>.<ext>, where the extension <ext> is one of mvt, png, jpg, webp, avif.

Remote URLs

Cloud storage URLs can be any URL recognized by gocloud. Configure a custom endpoint and region:

s3://BUCKET_NAME?endpoint=https://example.com&region=REGION

You may need to escape special characters like & and ? in your shell.

Cloud Storage Permissions

To upload your files to AWS S3 you will need an IAM policy for writing/reading to a bucket, at minimum this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::my-bucket-name/*"
        }
    ]
}

go-pmtiles's Issues

Add `pmtiles extract` command

It would be useful to be able to extract tiles from a pmtiles file for a certain area and put the extracted tiles into a new pmtiles file.

It could look something like this:

pmtiles extract input.pmtiles output.pmtiles --bounds "47.5,7.5,48.0,8.5" 

A usecase could be to have a global tileset and allow users to download an extract for a city or so.

Docker image

Put go-pmtiles docker image on GitHub registry

Extract command error

I've downloaded the sample 2.4gb level 10 zoom map and wanted to try and make one that only zooms to level 8
I'm trying to run the command to extract like this on a windows command line

pmtiles.exe extract --maxzoom=8 z10.pmtiles z8.pmtiles

But I just get the error

panic: extract <input> <output>

goroutine 1 [running]:
main. Main()
        /Users/bdon/workspace/protomaps/go-pmtiles/main.go:166 +0xc1e

Is there anything I'm doing wrong there, note that /Users/bdon/workspace is not my directory location, so not sure where that's coming from

convert/upgrade does not handle PNG/JPEG/WebP archives

• Disable the compressor
• Add an option to disable deduplication (requires prior knowledge)
• num tile contents data needs to be deduplication-aware
• Fix header values and reading MBTiles/PMTilesv2 metadata

Problem compressed vector tiles in MBTiles

Used go-pmtiles to convert a MBTiles with compressed vector tiles. It did not work in maplibre. Then I made the MBTiles with uncompressed vector tiles and converted again. Then it worked just fine. Am I doing something wrong?

implement gzip compression of tile data

This resolves one of the main drawbacks of PMTiles, which is that they can't have arbitrary Content-Encoding compression applied when delivered directly to the browser.

Unusable pmtiles file generated if source mbtiles file has no metadata

I think if you use the pmtiles command line tool to convert an mbtiles into a pmtiles file, and the mbtiles doesn't have bounds/center metadata, then you get a successfully-generated pmtiles file which the JS library (at least) will silently ignore.

I'm 80% sure this is what's happening, but I don't have the time to dig into the code to confirm at the moment (sorry).

serve error: 404 path mismatch: archive is type MVT (.mvt)

I spent the weekend trying to get a demo of v3 PMtiles up and running, but ran into an error with my own encoded pmtile.

Here is the pipeline I’ve developed:

osm.pbf -> tilemaker (with openmaptiles config) -> osm.mbtiles -> go_pmtiles -> osm.pmtiles

tilemaker --input hawaii-latest.osm.pbf \
--output hawaii.mbtiles \
--config config-openmaptiles.json \
--process process-openmaptiles.lua
pmtiles hawaii.mbtiles hawaii.pmtiles

This produces the following v3 .pmtile:

$ pmtiles show hawaii.pmtiles 
pmtiles spec version: 3
total size: 14 MB
tile type: Vector Protobuf (MVT)
bounds: -179.848300,10.835750 -137.522800,29.096240
min zoom: 0
max zoom: 14
center: -158.685550,19.965995
center zoom: 7
addressed tiles count: 5189
tile entries count: 5066
tile contents count: 4894
clustered: true
internal compression: 2
tile compression: 2
maxzoom 14
minzoom 0
name Tilemaker to OpenMapTiles schema
type baselayer
vector_layers <object...>
version 3.0
description Tile config based on OpenMapTiles schema
format pbf

Which I serve:

$ pmtiles serve .
2023/02/13 21:25:46 main.go:118: Serving  . on port 8080 with Access-Control-Allow-Origin: 

And then request a pbf over Hawaii with the following url:

http://localhost:8080/hawaii/10/301/384.pbf

The logs print:

2023/02/13 21:26:44 server.go:139: fetching hawaii 0-16384
2023/02/13 21:26:44 server.go:180: fetched hawaii 0-0
2023/02/13 21:26:44 main.go:115: served /hawaii/10/301/384.pbf in 1.64375ms

But in the browser I get a 400 error path mismatch: archive is type MVT (.mvt).

Any idea what that error means? Does the pipeline and process look correct?

Additionally, which frontend codebase is best for testing v3 pmtiles:

• PMTiles
• protomaps.js
• protomaps-themes

How to compile

How can I compile and run go-pmtiles on Ubuntu? I would like to mess around a bit with the source code...

Go panic when running convert mbtiles to pmtiles

Here are the logs:

2023/04/25 05:38:45 convert.go:362: # of addressed tiles:  5147885
2023/04/25 05:38:45 convert.go:363: # of tile entries (after RLE):  3117056
2023/04/25 05:38:45 convert.go:364: # of tile contents:  2840164
panic: runtime error: index out of range [3117056] with length 3117056
goroutine 1 [running]:
github.com/protomaps/go-pmtiles/pmtiles.build_roots_leaves({0xc021480000, 0x2f9000, 0x35e2aa}, 0x1000)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/directory.go:287 +0x31f
github.com/protomaps/go-pmtiles/pmtiles.optimize_directories({0xc021480000, 0x2f9000, 0x35e2aa}, 0x3f81)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/directory.go:310 +0xad
github.com/protomaps/go-pmtiles/pmtiles.finalize(0xc000186400?, 0xc00011a2a0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/convert.go:376 +0x30e
github.com/protomaps/go-pmtiles/pmtiles.ConvertMbtiles(0xc00063fdc8?, {0xc000124570, 0x24}, {0xc000045ec0, 0x3b}, 0x0?, 0x4?)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/convert.go:353 +0xfa8
github.com/protomaps/go-pmtiles/pmtiles.Convert(0x0?, {0xc000124570?, 0x24?}, {0xc000045ec0?, 0x3b?}, 0x7?, 0x58?)
/Users/bdon/workspace/protomaps/go-pmtiles/pmtiles/convert.go:110 +0x97
main.main()
/Users/bdon/workspace/protomaps/go-pmtiles/main.go:150 +0xb19

I'm currently working on isolating the problem on my end (it could be a malformed mbtiles archive), but at the very least this error should be actionable. The index out of range .. of length is the same number, so maybe there's an off by one bug somewhere? Has anyone run in to this problem?

return proper 403 or 404 errors when tiles are not found

Configure port number

I'll admit up front that I am least familiar with Go apps generally, so please excuse any very basic misses here. I love the concept of PMTiles and hope to test, initially focusing on python, but it seems this Go server is the more robust option.

I had hoped to take a shortcut to free deployment via Heroku for testing (following this workaround for compiled app https://medium.com/ki-labs-engineering/deploying-a-native-go-binary-on-heroku-6d4c955819d8), but the lack of configuration of port seems to be the main blocker. There appears to be no way to utilize a dynamic port currently.

Please disregard if this is an edge case, or if there is generally a better way to approach deployment of a precompiled Go app such as this. Excited to figure this out...

pmtiles extract feedback thread

Comment in this thread with your reports on version 1.9's extract command.

HOW TO USE

pmtiles extract https://r2-public.protomaps.com/protomaps-sample-datasets/protomaps-basemap-opensource-20230408.pmtiles out.pmtiles --region myregion.json

• works on public HTTP pmtiles, local files, S3-compatible buckets, Azure blobs, Google Cloud Storage buckets
• Input region can be any GeoJSON - Polygon, MultiPolygon, Feature, FeatureCollection

Info

• Your source extract size
• Your query region and maxzoom
• Your source bucket relative to your location (EC2 to S3 in same region, Cloudflare R2 to desktop across ocean, etc)
• Information printed in the run of pmtiles extract

example run, extracting z0-z15 US + Mexico from a planet OSM basemap hosted on Source Cooperative, medium EC2 instance in same AWS region

./pmtiles extract protomaps/openstreetmap/tiles/v2/protomaps-basemap-opensource-20230408.pmtiles us_and_mexico.pmtiles --bucket=s3://us-west-2.opendata.source.coop --region=us_and_mexico.json
WARNING: extract is an experimental feature and results may not be suitable for production use.
fetching 303 dirs, 303 chunks, 15 requests
Region tiles 26196418, result tile entries 15282945
fetching 15282945 tiles, 4261 chunks, 339 requests
fetching chunks 100% |█████████████████████████████████████████████████████████████████████████████████████| (17/17 GB, 33 MB/s)
Completed in 9m24.19785483s seconds with 1 download thread.
Extract required 357 total requests.
Extract transferred 18 GB (overfetch 0.1) for an archive size of 17 GB
Verify your extract is usable at https://protomaps.github.io/PMTiles/
Feedback wanted! report your success or failure to https://github.com/protomaps/go-pmtiles/issues

That's 28,000 tiles a second

Right now the download is unoptimized and single threaded - the priority right now is to ensure that the result is correct, we'll make a multithreaded extractor after.

Things to try

• pass the option --overfetch=0.1 By default, the downloader will batch nearby requests by downloading slightly more data, by default 10%. --overfetch=1.0 lets it download twice the final size of data.
• --dry-run prints out stats without actually downloading tile data

[feature request] bulk import tiles (fresh & updated ones)

feature request

I'm pre-rendering millions of raster-files on multiple processes. Since sqlite-based databases still aren't any good on this, I'm directly writing them to the filesystem. It would be nice to bulk import fresh & updated tiles.

example

pmtiles import

1. When neither the pmtiles file, nor the tile doesn't exists, create a new pmtiles file and add it to the pmtiles file.
2. When pmtiles fle exists, but tile doesn't, add it to the pmtiles file.
3. When the tile already exists, overwrite the existing one.

note

I can try to do it on my own and contribute, but I'm not into GO.

Better optimization around 16,384 byte root directory threshold

If the directory does not fit in only the root, right now the code defaults to creating a root with leaves of a fixed size 4, 8, 16... kB.

It should be smart enough for two cases:

• If the total directory is slightly larger than 16,384, the root should have a mix of tile entries and leaf pointers
• If the total directory is much larger, it should try to fill as much of the 16,384 as possible.

Failed strconv.ParseFloat: parsing " 34.30084411"

When trying to convert mbtiles an error occurs:

main.go:159: Failed to convert xxx.mbtiles, Failed to convert MBTiles to header JSON, strconv.ParseFloat: parsing " 34.30084411": invalid syntax

The thing is that in the metadata the value for center was 28.33709717, 34.30084411, 18, i.e. contained spaces between values.

ETags/Retry support for "pmtiles serve"

Requires using https://gocloud.dev/concepts/as/

Use v3 as a library

I was using the writer interface of v2 combined with github.com/paulmach/orb to create my pmtiles file.

	writer := pmtiles.NewWriter("./base.pmtiles")
	for z := minZoom; z <= maxZoom; z++ {
		covering := tilecover.Bound(bound, z)
		for tile := range covering {
		    layers := mvt.Layers{.....layers.....}
		    tileData, _ := mvt.Marshal(layers)
    
		    writer.WriteTile(
			    pmtiles.Zxy{
				    Z: uint8(tile.Z),
				    X: tile.X,
				    Y: tile.Y,
			    },
			    tileData,
		    )
		}
	}

	metadata, err := json.Marshal(pmtiles.Metadata{
           ...metadata...
         })
	if err != nil {
		return err
	}
	writer.Finalize(metadata)

Is something similar still possible using v3?

To create a better separation of concerns, it might be a good idea to split the pmtiles folder. The pmtiles folder can hold all the library code for reading and writing pmtiles. Next to a cmd folder that can hold the cli related commands and logic for handling flags, etc.

Cannot specify location of temporary file for convert

If your /tmp is mounted on a separate partition that doesn't have enough space to hold the pmtiles, the conversion of a mbtiles can never succeed.

convert does not fail when disk space runs out

should show an error

feature: pmtiles download

like extract, but the whole thing.

Show file spec version

/pmtiles show file:// public/catch.pmtiles 
total size: 13 MB
tile type: Vector Protobuf (MVT)
bounds: 140.962409,-39.136654 149.974995,-33.981390
min zoom: 0
max zoom: 15
center: 142.476196,-38.371809
center zoom: 15
addressed tiles count: 322258
tile entries count: 55994
tile contents count: 42135
clustered: true

Surprised it doesn't list what spec version the .pmtiles file is - that would be helpful.

[v1.7.0] panic on `verify `

error output

./pmtiles verify xyz.pmtiles
panic: verify <input>

goroutine 1 [running]:
main.main()
        /Users/bdon/workspace/protomaps/go-pmtiles/main.go:166 +0xc19

version

pmtiles 1.7.0, commit dfb4fe68e5c6b25eec8703864863fcefb2effb3d, built at 2023-02-06T03:33:16Z

Feature: pmtiles stats

• coordinate with @msbarry on a rough CSV format that can be quickly turned into DuckDB
• needs to be produced in a single pass
• come to consensus on how duplicates are treated

Downstream requirements:

• joining with ("top 100k statistics")
• interactive visualization. We could write a very small web server utility that runs off the DuckDB file and spits out vector tiles or raster images: OR, we could create a drag-and-drop UI to run DuckDb in WebAssembly and produce tiles via addProtocol

macOS notarization

feature: patch header/metadata from JSON

define a JSON schema like

{
  'tile_type':0,
  'metadata':{
    'my_key':'my_value'
  }
}

to allow modification of pmtiles by writing a new file.

pmtiles show --header-metadata-json would also be useful to output this.

List available commands

./pmtiles 
2022/10/27 14:16:01 main.go:22: Command is required.

There doesn't seem to be any way to get the available commands listed. ./pmtiles --help doesn't work.

add "show z x y" command and "replace metadata" command

feature: pmtiles verify

This should work against a local file only, verifying that:

• the clustered invariant holds
• tiles are contained within the file
• if gzipped, each referenced tile has the gzip magic bytes
• statistics are correct

Etc...

allow converting tile types

It'd be great to allow converting between any pair of supported tile types (mvt, png, jpeg, or webp).

Maybe this could be an option for pmtiles convert --format png <a> <b>?

Here's what happened to me (and I imagine other users not super familiar with mapping tools can easily run into similar problems):

1. I downloaded a small map
2. I set up hosting following the Cloudflare instructions
3. Since https://protomaps.com/docs/frontends/leaflet didn't have React-based instructions, I followed a pretty standard Leaflet setup with react-leaflet, pointing a TileLayer at https://<mydomain>/<pmtiles name>/{z}/{x}/{y}.mvt. This resulted in a CORB error due to the fact that TileLayer was creating img objects with mvt files that the Cloudflare server was serving with application/x-protobuf.
4. I ended up messing around too long with CORS stuff before realizing that the CORB issue was due to the content type.
5. After more tinkering I figured out how to use the protomaps npm library to create the layer from mvt files.

There are a few options that would make this easier for new users:

1. Allow converting tile types with the CLI
2. Allow selecting a specific format when downloading small maps
3. Add documentation for react-leaflet (I created a separate issue for this)
4. Add conversion support at the Cloudflare worker level

Caddy plugin

Initial implementation here: https://github.com/protomaps/go-pmtiles/blob/main/caddy/pmtiles_proxy.go

Outstanding tasks:

• The plugin appears at https://caddyserver.com/download when I register the package github.com/protomaps/go-pmtiles. It fails to make a build when I click download.
  • I assume this is because my plugin is in the package 'pmtiles_proxy', a subdirectory of this repository; the go.mod is at the root of the repo, and is the module also for the CLI program.
  • I would prefer not to maintain a separate git repository or separate go module go.mod just for Caddy. I would also prefer not to add caddy as a dependency of go-pmtiles (unless the parts needed by the plugin are extremely lightweight) how should I re-organize this repo?

here is an example Caddyfile:

localhost:2019 {
  route /* {
    pmtiles_proxy {
      bucket https://example.com
      cache_size 64
    }
  }
}

A request for https://localhost:2019/foo/0/0/0.mvt will fetch https://example.com/foo.pmtiles. Great!

• This only works because we have route /*. If we did route /prefix/*, we need to pass this information into the plugin so it can strip that off when it makes a fetch to the pmtiles file. Is this configuration state something I can access in Provision via caddy.Context? This overlaps with #61
• We also need the localhost:2019 information in order to generate correct TileJSON. I hardcoded this to https://example.com for now. Is this something we can access in Provision or do we need to trust the host of the request struct?

feature: pmtiles download

like extract, but the whole thing.

pmtiles upload is unreliable for large files

Look into S3 retries

writer improvements

• support leafdir writing
• support deduplication
• guarantee Hilbert optimization

Convert v3 produces corrupted data at zoom level 9+

Underlying data is OK but upload is producing corrupted archives past a certain point compared to rclone

API refactor

Hi first thanks for this project.

I'm moved my project from mbtiles to a dedicated kv solution and finally to pmtiles which is way better, problem is the actual go-pmtiles is not directly usable as an API.

I had to copy some private part of your code, to create a different http handler https://github.com/akhenakh/kvtiles/blob/master/storage/pmtiles/pmtiles.go

Are you planning on making a more Goish API?

Add `pmtiles` to homebrew for easy installation - needs more GitHub stars and watchers

Makes it much easier to keep things up to date

https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request
https://docs.brew.sh/Formula-Cookbook
Here's the tippecanoe formula: https://github.com/Homebrew/homebrew-core/blob/master/Formula/tippecanoe.rb

Since this is open source it should be acceptable in homebrew-core: https://docs.brew.sh/Acceptable-Formulae

edit: looks like there's even a tool for this: https://jkawamoto.github.io/fgo/ and goreleaser also can generate a formula: https://goreleaser.com/customization/homebrew/

feature: efficient extract of zoom level subset

This should "just work" with one request, but verify it works

Improved Documentation

Hi,

Please could you add examples of how to use pmtiles via docker to build or upload pmtiles

Beleive it something like docker run -v $(pwd)/data:/var/ ghcr.io/protomaps/go-pmtiles:main convert radon.pmtiles radon.mbtiles, however this doesn't work for me.

Please could you also provide more detailed instructions to install on linux - I can see the binary but don't know how to install

I'll happily add to the readme and submit a PR once I know how.

Kind regards,

Edd

Catch more invalid files with verify command

I'm experimenting with generating pmtiles (with MVT) from Rust. https://github.com/dabreegster/od2net/blob/tippecanwho/tippe/src/main.rs is my initial attempt. It doesn't produce usable output yet, but I'm having trouble figuring out where the problem is.

https://www.dropbox.com/scl/fi/25bikp6xvb3l6we182zt9/bad_output.pmtiles?rlkey=wn19lcqeramyxfv93jtmwdjt5&dl=0 is a bad output I'm producing right now. pmtiles verify shows no errors and seems happy. Loading the file with https://protomaps.github.io/PMTiles doesn't show anything on the map, and the console has an error:

e24e0201-5cdb-4bcb-8107-86ee3d56b440:2 Uncaught Error: unknown command 4
    at wc.loadGeometry (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:90342)
    at sa (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:64684)
    at f0.populate (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:100722)
    at C.parse (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:217921)
    at e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:220887
    at e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:220047
    at u.Actor.processTask (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:194145)
    at D5.process [as _callback] (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:193365)
    at D5.MessageChannel._channel.port2.onmessage (e24e0201-5cdb-4bcb-8107-86ee3d56b440:2:159810)

The results of pmtiles show:

pmtiles spec version: 3
tile type: Vector Protobuf (MVT)
bounds: 0.000000,0.000000 0.000000,0.000000
min zoom: 0
max zoom: 0
center: 0.000000,0.000000
center zoom: 0
addressed tiles count: 1
tile entries count: 1
tile contents count: 1
clustered: true
internal compression: 2
tile compression: 1
vector_layers <object...>

I'm pretty certain I'm failing to populate some of the metadata correctly, or doing something very wrong with encoding geometry. My ask here is if it makes sense for go-pmtiles verify to check for possible problems more.

ensure slashes work in pmtiles serve

V2 to V3 migrator: don't use min/maxzoom from metadata, derive from entries

Otherwise these will default to 0 if not present.

Path-related bugs on windows

related to backslashes.

convert: don't crash on empty mbtiles

[feature] Set cache headers

It seems that tiles are not cached (trying on latest chrome, linux)

Looking at others they all set cache-control:

• OpenStreetMap: cache-control: max-age=604800, stale-while-revalidate=604800, stale-if-error=604800
• OpenTopoMap: Cache-Control: max-age=439803
• Swisstopo: cache-control: public, must-revalidate, max-age=3600
• IGN: access-control-max-age: 43200
  etc

Combine pmtiles files

I'd like to combine two (or more) .pmtiles files. Browsing the docs I don't see a way to do this.

As a workaround I also considered converting the .pmtiles files back to .mbtiles files and using https://github.com/mapbox/tippecanoe 's tile-join tool to join them into a single .mbtiles file, and then back to a .pmtiles file. But I don't see a way to convert .pmtiles to .mbtiles.

Let me know if I'm missing something. Otherwise its just a feature request. Thank you!

Feature Request: Env variables to set CLI opts

It'd be great if I can configure options using environment variables to have deployments be easily managed via docker, etc.

PMTILES_PATH=https://storage.googleapis.com/my-tiles
PMTILES_CORS=*
PMTILES_CACHE=2000

404 with some (big?) pmtiles

TL;DR I can't get tiles out of go-pmtiles v0.5 for some files which work well through pmtiles-js.

I suspect it is linked with the file having leaf directories.

File	Size	JS - test URL	GO - test URL
Bugianen_Cervino	0.2 GB	🆗	🆗 wtracks C-go
Bugianen	3.5 GB (sorry!)	🆗 wtracks B-js	⛔ wtracks B-go
eslo14_walps	1.5 G	🆗	⛔

Context

I am integrating some pmtiles maps in https://opoto.github.io/wtracks/

After merging pmtiles support (opoto/wtracks#21), I have working, but excruciatingly slow pmtiles-js-on-HTTP1 maps (that we discussed in protomaps/PMTiles#16). For example IT Bugianen (find it in More... menu or though the custom link above)

Now I'm trying to evaluate performance on go-pmtiles on the same map, but it does not work (metadata works 404 on tiles I tried).
See the links to download/test the files above. I including a valid file to show that the issue is not how my server setup on montagne.top.

To test on wtracks, search for a covered area for both maps eg "Aosta".
Or test with curl:

curl -i https://www.montagne.top/maki/Bugianen_Cervino/16/34100/23382.jpg
HTTP/1.1 200 OK
curl -i https://www.montagne.top/maki/Bugianen/16/34100/23382.jpg
HTTP/1.1 404 Not Found
curl -I https://www.montagne.top/tile/eslo14_walps/16/34100/23382.png
HTTP/1.1 404 Not Found

I can try to debug it myself at some point, but figured maybe it's a known issue with big files?

Missing row issue

How do i start to debug a 'Missing row' error? Not much information is returned.

main.go:154: Failed to convert /home/ubuntu/tilemaker/planet.osm.mbtiles, Missing row

I'm using v1.8.0, with tilemaker v2.4.0

My script to generate the planet is....

aws s3 cp s3://osm-pds/2023/planet-230529.osm.pbf /data/planet-latest.osm.pbf
wget https://osmdata.openstreetmap.de/download/water-polygons-split-4326.zip -O /home/ubuntu/tilemaker/water-polygons-split-4326.zip
unzip -o water-polygons-split-4326.zip
ln -fs water-polygons-split-4326 coastline
wget https://osmdata.openstreetmap.de/download/land-polygons-complete-4326.zip -O /home/ubuntu/tilemaker/land-polygons-complete-4326.zip
unzip -o land-polygons-complete-4326.zip
ln -fs land-polygons-complete-4326 landcover
tilemaker --input /data/planet-latest.osm.pbf --output planet.osm.mbtiles --skip-integrity --store /tmp --compact
go-pmtiles convert planet.osm.mbtiles planet.pmtiles

Feature: PMTiles covering

Visualize a covering on a map