Performs a full MapOsMatic installation in an Ubuntu "Focal" 20.04LTS VM using Vagrant and shell provisioning
- Components
- Requirements
- Installation and useage
- Startup messages
- File system layout
- Keeping the data up to date
- Adding a new style or overlay
- Provision a real server
The following components will be installed into the VM:
- OCitysMap rendering backend (from my own modified fork)
- MapOSMatic web frontend (again from my own fork)
- Stylesheets:
- Current OSM default style
- MapOSMatic Printable style
- HikeBikeMap style
- Humanitarian style (HOT)
- MapQuest European style
- German openstreetmap.de style
- ... and more ...
- Overlay stylesheets:
- Golf overlay
- Fire Hydrant overlay
- ... and more ...
- A working Vagrant (>= v2.2) / Virtualbox (>= v5.2) setup
- The
vagrant-vbguestplugin to automatically install the host's VirtualBox Guest Additions on the guest system, install withvagrant plugin install vagrant-vbguest(only for VirtualBox Vagrant's provider). - The
vagrant-disksizeplugin to grow the VMs disk space, install withvagrant plugin install vagrant-disksize - Optional: The
vagrant-cachierplugin, to cache apt, gem, npm, pip packages, install withvagrant plugin install vagrant-cachier - A minimum of 4GB available RAM, 3GB for the VM, and 1GB extra head room for the host system
- About 30GB of disk space minimum (the more the larger your OSM PBF extract import file is)
- A working internet connection
- Sufficient bandwidth, about 4GB of data will be downloaded during installation and provisioning
- Copy a OSM PBF extract of your choice into this directory. If multiple files with ending '.pbf' are found only the first one is used.
- Run
vagrant up - Be patient ...
- The stylesheets require quite some extra downloads, and some processing on these (shape files, height information, ...). The downloads are cached localy, so downloads only happens on the first start mostly
- Importing the provided OSM PBF file can take some time, too, depending on its size ...
- When the VM starts to test the different style sheets and overlays it is actually already ready to use.
- You can access the web interface on http://localhost:8000/
- Or you can log into the VM with
vagrant ssh, e.g. to run the command line renderer directly or to do actual development work
The provisioning scripts will produce a lot of output when the VM is started for the first time.
Everything that is printed in green is just progress info and can easily be ignored.
Red output may be more serious, but unfortunately some tools print their progress messages to standard error output, too, so some red output can't be avoided.
Red things that can be ignored are:
- everything starting with
NOTICE: - lines starting with
warning:in the "pghyghtmap" secition - everything in the "DB IMPORT" section, unless there's an actual error messages at the end of the red block
ERROR: "planet_osm_point" is not a table or materialized viewin the "OpenRailwayMap" style block- everything in the "WAYMARKED" section as long as all data up to
Importing slopes DBgets processed - npn
WARNmessages in the "DJ FRONTEND" section - everything in the 'OSM_Notes_Overlay' section for now (it needs fixing on my side, but won't affect the other map styles
- Vagrantfile
- Main virtual machine setup file
- provision.sh
- Top level provisioning script, executes all scripts found in the inc/ folder
- cache/
- Used for cacheing downloads
- files/
- Contains extra files needed by the provisioning scripts
- inc/
- Contains all the provisioning shell scripts
- inc/styles/
- Provisioning scripts and ocitysmap ini file snippets for map styles
- inc/overlays/
- Provisioning scripts and ocitysmap ini file snippets for map overlays
- test
- Shared folder for running render tests in the VM and have results visible on the host
Inside the VM almost everything gets installed under the /home/maposmatic directory.
- elevation-data
- Ditigal elevation model data for hillshadings and reliefs
- ocitysmap
- The MapOSMatic OCitysMap renderer
- maposmatic
- The MapOSMatic web frontend
- osmosis-diffimport
- Data dir for OSM diff imports to update the databasse
- shapefiles
- All shapefiles required by the different map styles installed
- styles
- All installed mapstyles are here, with the exception of the maposmatic-printable style which is part of the ocitysmap renderer repository
- tools
- All tools that need to be installed from source
- umgebungsplaene
- The neighbourhood maps project, an alternative frontend to the MapOSMatic rendering service
If the OSM PBF file you used for the initial data import provides a replication base URL to fetch diffs from, a systemd service will be set up to download such diff files and to apply the changes to the database.
E.g. GeoFabrik provides daily diff files for all their regional extracts, so if you downloaded the PBF file used for initial setup from there your databases can be brought up to date with
systemctl start osm2pgsql-update.service
systemctl start waymarked-update.serviceIf you want to import updates on a daily basis automatically you can enable the systemd timer that also got installed for this service:
systemctl enable osm2pgsql-update.timer
systemctl start osm2pgsql-update.timer
systemctl enable waymarked-update.timer
systemctl start waymarked-update.timerThis will run the diff update services once per day each, or whenever the VM is restarted.
If no replication_base_url information is found in the initial import
file, then the service unit and timer will not be installed at all.
(This unfortunately is also true for full planet files at this point, but I don't expect anyone to try a full planet import inside a VM anyway. If you actually do plan to do this, please let me ([email protected]) know and I'll see what I can work out to support automatic diff import setup for this, too)
Also the shapefiles can be updated every once in a while. This does not depend on the import file used, so the service and timer files for this are always installed. You can start the shapefile update service manually with
systemctl start shapefile-update.serviceor enable daily automatic updates with
systemctl enable shapefile-update.timer
systemctl start shapefile-update.timer- needs to be Mapnik XML, or a format that can be converted into that, like CartoCSS
- needs to use the "osm2pgsql" database schema (e.g. the "imposm" schema is not supported)
The setup uses the --hstore-only approach to import the OSM data into PostGIS,
meaning that all OSM attributes are stored in a hstore column tags only. Most styles
expect to have explicit feature columns in tables like planet_osm_polygon though.
To support this, and to be able to add new feature columns on short notice withough
having to rebuild tables, the hstore-only tables created at import time are actually
named planet_osm_hstore_polygon etc.
The actual planet_osm_% tables are then implemented as views instead, mapping specific
tags hstore entries to view columns. This model is used by the German OSM style by
default, and can easily be adapted to other styles, too, as the fact that they are
actually reading from views and not from the actual tables is totally transparent to
them.
If the style you want to add uses feature columns not present in the views yet, you can easily add them to the view declaration and re-import just that.
E.g. if you need to add a column named atm to the planet_osm_point table
for a banking related style, open the files/database/db_views/planet-osm-point.sql
file, and add this at the end of the view definition, right above the FROM line:
CREATE OR REPLACE VIEW planet_osm_point AS
SELECT osm_id
[...]
, tags->'wetland' as "wetland"
-- after initial import add further columns below this line only
, tags->'atm' as "atm" -- <-- this line added by you
FROM planet_osm_hstore_point;
[...]The inside the VM, run:
sudo -u maposmatic psql gis < /vagrant/files/database/db_views/planet-osm-point.sqlto modify the view.
Adding at the end is necessary when replacing the view, as there may already be other views added by other styeles that rely on this views column order. When you add your new column before the initial import, or plan to do a re-import, you can as well add the new column at the right sorting position right away.
Shapefiles are stored under /home/maposmatic/shapefiles/ to avoid duplicate
download bandwidth and storage space.
Try to create symlinks from there to the path your style expects the shapefiles in, or modify the paths in the style righ away to point to the central shapefile directory.
If your style needs a shapefile not yet present, consider adding it to the
inc/get-shapefiles.sh script by adding a line like
URLS+="http://example.com/path/to/some-shapefile.zip"and re-run the script with
/vagrant/inc/get-shapefiles.shThis will check for all shapefile archives that have not been downloaded yet,
or which have change since last downloaded, to download and install these under
/home/maposmatic/shapefiles/
You can also use the provisioning scripts from this project to set up a real server instance instead of a Vagrant VM, although it's still a little bit 'hacky' to do so. Also this will only work on a Ubuntu 18.04LTS system.
As for now the /vagrant and /home/maposmatic base paths are still
hard coded into several of the provisioning scripts you have to check out
the project into directory /vagrant, put a PBF file to import there,
and then run the master provisioning script manually:
git clone https://github.com/hholzgra/maposmatic-vagrant/ /vagrant
cd /vagrant
... copy, symlink or download .osm.pbf file to this director ...
bash provision.sh
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent