Problem with docker image about osmtilemaker

[audour]

Thanks a lot for your feedback ! Good to know that you were able to complete the process.
And ok for your docker options, good to know.

About --flat-nodes, sorry if it was not enough visible but it was said in conf/config :

# uncomment next line if you want to use --flat-nodes (only for huge extracts like planet or europe, see README.md)
# OSM2PGSQL_OPTS="$OSM2PGSQL_OPTS --flat-nodes $DOCKERPATH_OSM2PGSQL_FLATNODE_DIR/nodes.cache"

Maybe we should add it directly in the README.md.

[audour]

Hi,
I can't reproduce with Docker version 20.10.5 under Ubuntu 20.04.2 LTS.
Maybe try to run the basic postgres:10 docker image from https://hub.docker.com/_/postgres to see if it works ... it's the base image of osmtilemaker.

[abmancini]

Thanks for the response.
Actually in my setup I had to delay the installation of postgis.

I commented out the 2 lines in Dockerfile
postgresql-${PG_MAJOR}-postgis-${POSTGIS_VERSION}
postgresql-${PG_MAJOR}-postgis-${POSTGIS_VERSION}-scripts
and installed them later in the interactive section just after
sudo docker exec -it --workdir=$DOCKERPATH_SOURCE_DIR $DOCKER_NAME /bin/bash

Docker version 20.10.5, host os CentOS Linux 7

Eventually, as soon as I understand if all works well in my configuration, I close this issue documenting all the changes i did.

[audour]

Nice if you've found a workaround, and perfect if you can share it

[audour]

Hello @abmancini
Finally did it work as expected ? I hope it did.

[abmancini]

Hello,
sorry for the long response time but we had some major hardware problem and I had to restart the whole process and finally I had success in generating tiles for the whole planet.

All the process was quite smooth but I had to adjust some of the commands (mainly cause the used version of docker and Linux flavor); follows a note about all the changes made.

Docker Version: 1.13.1
Host OS: CentOS Linux 7
OS/Arch: linux/amd64
Size of planet-latest.osm.pbf: 59GB

Executing the docker image:

sudo docker run -d --privileged \
  --env POSTGRES_PASSWORD=$DBPG_USER_POSTGRES_PWD \
  --env DBPG_USER_OSMTILEMAKER_USERNAME=$DBPG_USER_OSMTILEMAKER_USERNAME \
  --env DBPG_DATABASE_NAME=$DBPG_DATABASE_NAME \
  --volume $(pwd):$DOCKERPATH_SOURCE_DIR \
  --volume $HOSTPATH_WORKING_DIR:$DOCKERPATH_WORKING_DIR \
  --volume $HOSTPATH_OSM_FILE_DIR:$DOCKERPATH_OSM_FILE_DIR \
  --volume $HOSTPATH_PG_DATA_DIR:/var/lib/postgresql/data \
  --volume $HOSTPATH_OSM2PGSQL_FLATNODE_DIR:$DOCKERPATH_OSM2PGSQL_FLATNODE_DIR \
  --volume $HOSTPATH_TILES_DIR:$DOCKERPATH_TILES_DIR \
  --publish $DOCKER_HOST_PORT_TO_PUBLISH:5432 \
  --shm-size="5G" \
  --memory=50g \
  --name $DOCKER_NAME \
  $DOCKER_BUILD_TAG

i had to add: --privileged --shm-size="5G" --memory=50g

Starting the interactive bash session

in the command
sudo docker exec -it --workdir=$DOCKERPATH_SOURCE_DIR $DOCKER_NAME /bin/bash
unfortunately --workdir does nto works for me so i had to remove the option and manually chane directory once started the session.

Osm2pgsql import

This resulted to be far the most time consuming and problematic step.
The command that finally imported the whole planet file is the following

osm2pgsql --username osmtilemaker --database osmtilemaker --hstore \
--style /docker_mounted_volumes/working_dir/openstreetmap-carto/openstreetmap-carto.style \
--tag-transform-script /docker_mounted_volumes/working_dir/openstreetmap-carto/openstreetmap-carto.lua   \
--slim --drop --cache 11000 \
--flat-nodes  /docker_mounted_volumes/working_dir/mycache.bin 
--multi-geometry --multi-geometry /docker_mounted_volumes/osm_data/planet-latest.osm.pbf >> $DOCKERPATH_WORKING_DIR/osm2pgsql.log 2>&1 &

where i had to use the option --flat-nodes /docker_mounted_volumes/working_dir/mycache.bin to speedup the process.

Osm2pgsql took 228313 seconds (about 64 hours) overall and at the end the size of the pg_data directory is 357GB.

Thanks.

[audour]

About --flat-nodes, sorry if it was not enough visible but it was said in conf/config :
[...]
Maybe we should add it directly in the README.md.

Done through fe57cf6