openmaptiles / openmaptiles Goto Github PK
View Code? Open in Web Editor NEWOpenMapTiles Vector Tile Schema Implementation
Home Page: https://openmaptiles.org
License: Other
OpenMapTiles Vector Tile Schema Implementation
Home Page: https://openmaptiles.org
License: Other
@klokan Any feedback on mobile performance with the latest release v3.1?
There is an example of a water layer, but it might not be what everyone needs. It's not clear how to add two layers to the repo which cover the same content and have the same most-obvious name.
I'm looking at the place layer and have some feedback
class
(String): country
, state
, settlement
, subregion
, or other
.capital
(Numeric): For settlements the place is a capital, what admin_level it is capital of. 2
and 4
only.place_detail
(String): For settlement
, one of city
, town
, village
, hamlet
, isolated_dwelling
. For subregion
, suburb
or neighbourhood
. For other
, one of locality
or farm
.I'm not certain of rank
. As described, it ties an implementation into using Natural Earth data.
I would like to create some draft layers ( for personal interest and for creating basic examples )
First is in my mind a wheelchair
topic :
1 New layers + 1 New tileset definitions !
poi_wheelchair
layers
poi
layersIn my mind - it is :
Can I add this repo ?
๐ What is a (future proof) suggested method ?
Problems:
So we discussed this further.
We want to have the documentation in a website just like Mapzen https://mapzen.com/documentation/vector-tiles/layers/
There will be a distinction between "Cartographer" (layer schema) and "Developer" (diagrams made by @ImreSamu ) docs.
The README in the layer directory should not be there (since it is just duplicated code) , there should be a link to the website documentaiton however in the README.
When drawing extruded buildings there may be artifacting due to multiple overlapping geometries.
Likely resolution:
Adding 'DISTINCT' to the building.sql file.
Right now it is in the openmaptiles repo and you have to call make
.
It could also be in the website or wiki.
When rendering roads and rails it's necessary to handle ordering of both together - i.e. place them relative to the layer tag, not always have roads before or after rails.
This is done in tilezen and is necessary for rendering with Mapnik. osm-carto does it with a big UNION ALL that gets both roads and rails and allows for objects which are both.
housenumber_centroid.sql contains
UPDATE osm_housenumber_point SET geometry=topoint(geometry)
WHERE ST_GeometryType(geometry) <> 'ST_Point';
topoint is not defined anywhere.
If you have a feedback about POI layer - comment to this issue ...
SQL Output: (secret gist ) : https://gist.github.com/ImreSamu/a666974defc8c67da494349239b30045
based on 2016.nov.04 taginfo data and imposm3 mapping
( ~ select key,value, count_all from taginfo where not in [['imposm3_mapping']] order by count_all desc Limit 100 )
My rule of thumb :
My suggestion for the NEW POI-s based on the SQL output ..
key | value | count_all
---------+------------------------+-----------
amenity | drinking_water | 127332 ! https://github.com/mapbox/maki/blob/master/icons/drinking-water-11.svg
amenity | ice_cream | 8726 ! https://github.com/mapbox/maki/blob/master/icons/ice-cream-11.svg
shop | car_parts | 24385 !
shop | variety_store | 17220 !
shop | tyres | 10265 !
sport | football | 3418 !
sport | fitness | 2347 !
key | value | count_all
---------+------------------------+-----------
amenity | atm | 109383 ?
amenity | hunting_stand | 85403 ? ( for outdoor )
amenity | vending_machine | 82017 ? ( city maps)
amenity | social_facility | 47957 ?
amenity | clinic | 36680 ?
amenity | charging_station | 9624 ?
amenity | emergency_phone | 4828 ?
shop | funeral_directors | 7484 ?
shop | farm | 6554 ?
shop | seafood | 5851 ?
shop | kitchen | 5040 ?
shop | paint | 4756 ?
shop | trade | 4356 ?
shop | bookmaker | 3839 ?
shop | estate_agent | 3808 ?
shop | pawnbroker | 3756 ?
shop | lottery | 3680 ?
shop | houseware | 3511 ?
shop | fashion | 3208 ?
sport | 10pin | 2104 ?
sport | exercise | 1594 ?
sport | trampoline | 1226 ?
sport | softball | 1007 ?
sport | 9pin | 1005 ?
leisure | sauna | 5276 ?
leisure | fitness_centre | 4812 ?
leisure | fitness_station | 4387 ?
leisure | horse_riding | 3876 ?
edited:
The new vector tile schema v3.1 is based on making Positron and Dark Matter and it would be good to have someone read through the layer documentation and give feedback.
My ideal use case
layer=railway
)dataupdate=monthly
or hourly or never? )like this:
openmaptiles clear_all
openmaptiles import_defs github.com/cleartables/omt_layers/cleartables/clearrailway_slim layer=railway dataupdate=monthly
openmaptiles import_defs github.com/openmaptiles/layers/openmaptiles/building3d layer=building dataupdate=weekly
openmaptiles import_defs github.com/openmaptiles/layers/openmaptiles/landuse layer=landuse dataupdate=NO
openmaptiles import_defs github.com/klokantech/openvectordefs/klokantech/extendedpoi layer=poi dataupdate=daily
openmaptiles import_defs /myprivate_dir/layers/extremehiking layer=hiking dataupdate=daily
openmaptiles generate_jobs area=albania min_zoom=1 max_zoom=10 mbtiles_compress=4 parallelization=8 system=K8S update=yes
openmaptiles schedule_and_run_jobs
And if possible we can use imposm3
and/or osm2pgsql
tools in each layer
IMHO : Our FUTURE Problems ( based on my experience in working on big DW-s )
IMHO : we need to create and adapt some guideline, conventions for this ...
As I see we have 4 basic solutions for start of thinking ...
V1(PostgreSQL_Schema as a layer_group_id
)
V2(PostgreSQL_Schema as a layer_definition_id
)
V3(Table name prefix) based
public
schema and in inside every tables , the prefix separates themV4 mixed ( V1+V3 = grouping + table prefixing )
So I am thinking about this ...
Just as example how one could do that. From discussion #1
if we don't choose mapbox-gl.js which rendering component will be chosen?
I suggest either BSD-3
or MIT
for the code and something like CC0
for the styles.
railway=tram_stop
I'm inclined to think this should go with highway=bus_stop into its own layer.
This allows main lines to be distinguished from less important ones
See gravitystorm/openstreetmap-carto#1645 for an example, it makes a surprising difference.
render_height: An approximated height from levels and height of building after the method of Paul Norman in OSM Clear. For future 3D rendering of buildings
I've been skeptical about the use of this particular formula. I don't have it designed for complex 3d rendering, but simple 3d rendering, and I ended up pulling it from one of my styles because mapnik's building symbolizer doesn't work very well.
Before I lost my "first User Expericence" (UX) view :)
If possible ,
imho :
Now:
volumes:
- ./data:/import
- ./build:/mapping
or
- ./build:/sql
or
- ./data:/export
My First idea ( for brainstorming ) but not so easy to find a perfect naming standard ..
volumes:
- ./host_data:/data
- ./host_build:/build
Why?
For file management
but I am open for other suggestions ...
Wikipedia and @pnorman are working on a nice implementation of disputed borders which should make a way to this tile schema at some point.
Details in https://phabricator.wikimedia.org/T113008
Hello :)
My ideal README will contains only 3 line to test a new opensource project ..
https://github.com/openmaptiles/openmaptiles.git
cd openmaptiles
make quickstart
( or just a make start
or make bootstrap
or ... )
( draft, not tested - steps , based on the current documentation )
Why:
implementation:
Can we port something like bubblewrap to OpenMapTiles?
tangrams.github.io/bubble-wrap/
I started planet rendering yesterday evening for v3.1 mainly to test performance and the modified distributed. And also so we can put it on the CDN to test in the mobile apps.
The performance has increased a lot... right now it rendered half the planet in 12hrs with 6 machines.
With speeds on the workers ranging from 30 tiles/s to 80 tiles/s for each process (4 processes on each server). I guess the savings come from the table functions.
We have a ./quickstart.sh - for start new users ..
But the expected next questions
idea /draft specifications :
./create_tiles.sh < extract_id >
for downloading:
We can use https://github.com/julien-noblet/download-geofabrik
go get github.com/julien-noblet/download-geofabrik
go install github.com/julien-noblet/download-geofabrik
download-geofabrik update
download-geofabrik download albania
---> albania.osm.pbf
We need to extract the BBOX info from this
And we need to create a docker-compose config
PRO:
CON:
based on:
We have to find an optimal osm (geofabrik?) extracts
We have to pre-generate for an every extract a docker-compose extending yml file
~ expected output ...
## OpenMapTiles service definitions for zurich_switzerland !
version: "2"
services:
download-osm:
environment:
download_osm_url=https://s3.amazonaws.com/metro-extracts.mapzen.com/zurich_switzerland.osm.pbf
services:
generate-vectortiles:
environment:
BBOX: "8.25,46.97,9.58,47.52"
MIN_ZOOM: "0"
MAX_ZOOM: "7"
I'd open this in the CT repo, but this one is still private.
This is a checklist to make sure that ClearTables has the required content to produce the current layers. I've done the same thing with MB Streets, imposm default, and Geofabrik shapefiles to see if there's anything missing on the CT side.
As I see most of the POI data is missing ( compare to taginfo )
Probably the osm_poi_polygon
table is missing ? or we have to change the imposm3 mapping type ...
The excel in the Attachment is an example of a poi layer structure, do you think this is a different way to display the real word. if you think this is helpful I will enrich the structure.
Setup a quick webserver to demo the tile.mbtiles in quickstart.sh. I plan on using the existing mapbox studio classic.
In general length filters don't work because a waterway can be split into multiple parts without changing the meaning. When you use a length restriction, you get parts of the waterway not rendering.
As an example, in London much of the Thames is in sections under 1km long, and the z8 length restriction is 10km, which is 6% of a tile. Even closer to the water, the sections are still under 10km.
Start with ClearTables until we hit the wall. Then implement a table ourselves and then we have more technical experience with ClearTables.
I am still prototyping but already working with it and quite happy with the modular approach. Really rocks.
Example of water:
layers/water
โโโ mapping.yaml
โโโ water.sql
โโโ water.yaml
The layers/water/water.yaml
is the definition file of the layers.
It specifies docs, buffer size and the query and also which SQL files to get the schema from
and which imposm3 schema to use. In datasources
we might add different type
sources.
layer:
id: "water"
description: |
Water polygons and linestrings representing oceans, lakes and waterways.
buffer_size: 4
datasource:
query: (SELECT * FROM layer_water(!bbox!, z(!scale_denominator!))) AS t
schema:
- ./water.sql
datasources:
- type: imposm3
mapping_file: ./mapping.yaml
This definition files are then pulled together in a tileset
specification.
tileset:
layers:
- layers/boundary/boundary.yaml
- layers/highway/highway.yaml
- layers/ice/ice.yaml
- layers/building/building.yaml
- layers/state/state.yaml
- layers/country/country.yaml
- layers/place/place.yaml
#- layers/rail/rail.yaml
- layers/urban/urban.yaml
- layers/water/water.yaml
- layers/landcover/landcover.yaml
name: OSM2VectorTiles v3.0
description: "Free global vector tiles from OpenStreetMap. http://osm2vectortiles.org"
attribution: "<a href=\"http://www.openstreetmap.org/about/\" target=\"_blank\">© OpenStreetMap contributors</a>"
center: [-12.2168, 28.6135, 4]
maxzoom: 14
minzoom: 0
defaults:
srs: +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over
datasource:
srid: 900913
With the mapping containing the imposm3 mapping and generalized tables.
generalized_tables:
water_polygon_gen3:
source: water_polygon
sql_filter: area>9000000.0
tolerance: 350.0
water_polygon_gen2:
source: water_polygon
sql_filter: area>1000000
tolerance: 200.0
water_polygon_gen1:
source: water_polygon
sql_filter: area>500000
tolerance: 100.0
tables:
water_polygon:
fields:
- name: osm_id
type: id
- name: geometry
type: validated_geometry
- name: area
type: pseudoarea
- key: name
name: name
type: string
- name: name_en
key: name:en
type: string
filters:
exclude_tags:
- [ "covered", "yes" ]
mapping:
landuse:
- reservoir
natural:
- water
- bay
waterway:
- river
- riverbank
- stream
- canal
- drain
- ditch
type: polygon
water_linestring:
fields:
- name: osm_id
type: id
- name: geometry
type: geometry
- name: waterway
type: mapping_value
- key: name
name: name
type: string
- name: name_en
key: name:en
type: string
mapping:
waterway:
- stream
- river
- canal
- drain
- ditch
type: linestring
I switched to imposm3 now because it integrates much better with modular layers - actually switching the queries over from one schema to another is not much work.
I have a Python script now called omtgen.py
(for Open Map Tile generation tool) which takes care of generating tm2source, imposm3 mapping and aggregating all the SQL of all the layers.
@klokan Could you create a repo omtgen
where I develop the generation tool for OpenMapTiles? Better name for the CLI also welcome.
The generation tool must be able to do:
Not so fond of the repetition in the name.
Why not rename openmaptiles/openmaptiles-tools
to openmaptiles/tools
and openmaptiles/openmaptiles-gl-styles
to openmaptiles/gl-styles
?
๐
Imho :
What is the ideal github workflow for US ?
Testing requirements
Code conventions
....
Not critical, just a memento ..
Imho
Landuse from OSM data is often tricky, and people can decide to go different ways with it. This being said, I just can't figure out what the cartographic decisions behind the selections in https://github.com/openmaptiles/openmaptiles/blob/master/layers/landuse/landuse.sql are.
https://lists.osgeo.org/pipermail/postgis-devel/2016-October/025973.html
https://git.osgeo.org/gogs/postgis/postgis/pulls/5
This is very interesting - as it may save a lot of data transfers and reparsing between Postgres and Mapnik workers - which could mean a significant boost in the speed of the vector tile rendering (of the planet)
It plays well with the openmaptiles "layer-catalog" structure we have discussed - where tm2source is just one of the generated outputs. We could have a lightweight generator with core functionality done directly in SQL - especially for the microtiles sets.
Be able to label waterways (could stay on same layer). #20
testing ./quickstart.sh -> generate-vectortiles
docker-compose -f docker-compose.yml -f docker-compose-test-override.yml run --rm generate-vectortiles
I see lot of warnings .. Is it possible to remove ?
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer boundary)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer boundary)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer highway)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer highway)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer highway_name)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer highway_name)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer building)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer building)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer housenumber)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer housenumber)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer place)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer place)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer poi)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer poi)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer railway)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer railway)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer water_name)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer water_name)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer water)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer water)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer waterway)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer waterway)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer landcover)
Mapnik LOG> 2016-11-16 01:32:45: 'maxzoom' is deprecated and will be removed in Mapnik 4.x, use 'maximum-scale-denominator' instead (encountered in layer landcover)
Mapnik LOG> 2016-11-16 01:32:45: 'minzoom' is deprecated and will be removed in Mapnik 4.x, use 'minimum-scale-denominator' instead (encountered in layer landuse)
Create an analysis to check min, max, median of nodes and tilesize to ensure #31 will work well.
Produce graphs similar to
https://mapzen.com/blog/v1-vector-tile-service/
testing [new] ./quickstart.sh - I see lot of SQL warnings ..
cat quickstart.log | grep 'has type "unknown"'
psql:/sql/tileset.sql:952: WARNING: column "class" has type "unknown"
psql:/sql/tileset.sql:958: WARNING: column "class" has type "unknown"
psql:/sql/tileset.sql:964: WARNING: column "class" has type "unknown"
psql:/sql/tileset.sql:1062: WARNING: column "landuse" has type "unknown"
psql:/sql/tileset.sql:1062: WARNING: column "natural" has type "unknown"
psql:/sql/tileset.sql:1180: WARNING: column "landuse" has type "unknown"
psql:/sql/tileset.sql:1186: WARNING: column "landuse" has type "unknown"
psql:/sql/tileset.sql:1192: WARNING: column "landuse" has type "unknown"
The ID of the layer should be defined in the tileset not in the layer YAML.
This means you can pull in the layers as you want and still give them a custom name.
So not like this.
tileset:
layers:
- layers/boundary/boundary.yaml
- layers/highway/highway.yaml
- layers/highway_name/highway_name.yaml
- layers/building/building.yaml
- layers/housenumber/housenumber.yaml
- layers/place/place.yaml
More like this.
tileset:
layers:
boundary: layers/boundary/boundary.yaml
street: layers/highway/highway.yaml
streetname: layers/highway_name/highway_name.yaml
house: layers/building/building.yaml
housenumber: layers/housenumber/housenumber.yaml
place: layers/place/place.yaml
Because tilemaker is quite different than node-mapnik, it's a good example for including a layer definition that doesn't depend on node-mapnik.
A good example would be the demo tilemaker buildings and water layers.
From config.json
the layer definitions are
water:
{ "minzoom": 11, "maxzoom": 14 }
buildings:
{ "minzoom": 14, "maxzoom": 14 }
The metadata properties are
{ "id": "water", "description": "water", "fields": {}},
{ "id": "buildings", "description": "buildings", "fields": {}}
The proocessing lua is in https://github.com/systemed/tilemaker/blob/master/process.lua, but this covers both layers, as well as other layers.
Add continents to place
layer.
osm2vectortiles/osm2vectortiles#477
A lot of the layers have a query of the form
query: (SELECT geometry, class, subclass, properties::text FROM layer_railway(!bbox!, z(!scale_denominator!))) AS t
where layer_railway is a SRF which is only used in this one place. Speaking from experience, this is overuse of functions which will be a barrier to maintenance and understanding without offering gains.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.