OpenStreetMap about archives

[lukasmartinelli]

Since these companies offer tile servers as a paid service, I suspect they aren't going to be willing to share all their secrets :(

Creating vector tiles for the entire world is a big undertaking. At https://github.com/osm2vectortiles/osm2vectortiles we have a completly open process how to generate vector tiles for the entire planet but we dump everthing into a PostGIS database first.

We also looked at tilemaker but ruled it out for scaling to the entire planet. To scale up you need a lot of infrastructure as well (like 6 x 50GB RAM, 16 core servers).

I don't know exactly what the goal of this IPFS project regarding OSM is but we love to collaborate.
If you want to preserve data then vector tiles is perhaps the wrong format - it is not a loss less format.

We love to collaborate. We can also sit together in a Skype call (lukas.martinelli) to discuss questions regarding the entire vector tiles from OSM topic.

[whyrusleeping]

👍 this would be soo cool.

[zignig]

👍

[davidar]

Ok, I think we're basically looking for hoverboard by @devtristan, but loading all of its resources from ipfs. @mapbox and @mapzen have also published a lot of useful tools and data.

Relevant links:

• http://blog.mathieu-leplatre.info/serve-your-map-layers-with-a-usual-web-hosting-service.html
• http://gis.stackexchange.com/questions/125037/self-hosting-mapbox-vector-tiles

[jbenet]

👍 i would looooove love love this. I think we can do this very easily now. once ipld json-ness lands, this will be a trivial import (yay JSON!)

[davidar]

The major obstacle is converting the raw data dump into a format that the js viewers can read. I haven't had much luck with this yet, is anyone more familiar with osm able to lend a hand? :)

[davidar]

And the winner is tilemaker by @systemed :)

Here's a semi-functional OSM viewer on IPFS! It still loads some external js, but all the tiles are hosted on ipfs :)

• systemed/tilemaker#10
• patched luabind

Edit: The downside with tilemaker is that (according to the README)

It's best suited to city and region extracts

so, I still need to work out out to scale it up to the whole planet...

[jbenet]

@davidar this is really great! i've been waiting so long for this. 👍

[zignig]

interesting, have some go code for tile caches and osm stuff. will dig up.

[jbenet]

@davidar would be nice if the OSM viewer accepted a query params to specify things like the data root. (i've been using <url>#<ipfs-path> but # is a bad idea because it prevents regular anchor action. i now think ?somekey=<ipfs-path> is better

[zignig]

https://github.com/qedus/osmpbf is pretty good , i've used if before.

I think general GIS storage in IPFS will create some interesting options.

[davidar]

Hmm, from here about generating vector tiles:

We (in Nutiteq) spent many man-months to optimize it and to get it right for OSM data, and as far as I've heard from Skobbler and Mapbox guys they have done the same.

Since these companies offer tile servers as a paid service, I suspect they aren't going to be willing to share all their secrets :(

The most I've been able to find out about how mapbox generates their tiles is from a single slide in that video I linked earlier (@16min), transcript below:

# headless export

npm install mapnik mbtiles \
 tilelive tilelive-bridge

./node_modules/tilelive/bin/tilelive-copy \
  bridge:///Users/dane/data/world-borders.tm2source/data.xml \
  --bbox=-180,-85,180,85 \
  export.mbtiles

# data.xml is created by Mapbox Studio

The problem is that Mapbox Studio uses way to much memory for me to even consider trying to use it to generate worldwide tiles myself.

However, I also found @opensciencemap, who have a very cool vector tile viewer. As a a non-profit research project, hopefully they might be more open to collaboration. I'll try floating the idea with them.

Edit: Also @mapsforge, but it appears to be targeted at android

[whyrusleeping]

@davidar i have a few machines with a lot of memory, let me know if you want me to do anything for you

[tmcw]

@davidar happy to help out with understanding & creating vector tiles. The main design decision you need to make first on this task is whether you're creating vector tiles for people to use as basemaps, or as data. For instance, the Mapbox Streets tileset is for visualization, and the OSM QA tiles are for data and analysis. If you're creating tiles for visualization, they can't practically contain all data, and will need to bake in assumptions about what is shown at each zoom level and what map designs are appropriate. These assumptions vary between different maps, hence why there isn't one universal way to 'convert' OSM data to visualization-ready vector tiles.

[davidar]

@tmcw that would be brilliant :). My immediate interest is in visualisation (but I suspect @zignig would be very interested in the data analysis side). It would be great to have enough data to do things like routing, but that's not an immediate priority. I don't have any specific requirements other than:

• something that looks better than the crappy demo I made (linked a couple of comments ago)
• reasonable file sizes (preferably the total size should be not significantly larger than the raw osm data)

I'd very much appreciate it if you were able to offer guidance as to how to achieve this. Working with geo data isn't exactly my area (outside of geostats stuff like kriging, etc).

[almereyda]

@zignig We are investigating using IPFS as an archival option for geodata in @TransforMap. But this is not the OSM topic anymore and rather links to a world of distributed geodata. You are always invited for sidetracking to join our https://discourse.transformap.co

[jbenet]

@almereyda we'd love to help however we can. maybe open a different issue in this repo?

[davidar]

http://osm2vectortiles.org/

CC: @lukasmartinelli @manuelroth

[davidar]

@lukasmartinelli Glad to hear you're interested in collaborating :)

The goal of this particular issue is to get an OSM viewer running on IPFS. Lossiness isn't much of a concern as we can already just mirror the raw OSM dumps.

The major constraint is that all the tiles need to be stored in a static filesystem structure, without any reliance on server-side processing (ie. tileserver-php isn't an option). Is this likely to be problematic?

[lukasmartinelli]

The major constraint is that all the tiles need to be stored in a static filesystem structure, without any reliance on server-side processing (ie. tileserver-php isn't an option). Is this likely to be problematic?

If you serve vector tiles only you can extract and serve them on a static fileserver. There are examples that do vector tiles serving from S3 (but beware uploading 360 million files to S3 takes forever). You cannot do some features like supporting a filter zoom level or supporting other schemes like TMS but it works.

The goal of this particular issue is to get an OSM viewer running on IPFS. Lossiness isn't much of a concern as we can already just mirror the raw OSM dumps.

Generating vector tiles from the OSM dump just to get a mirror is quite a task. Probably better to not pregenerate all the vector tiles like we do it but serve them dynamically from the specific OSM dump.
Would be cool if tilemaker can handle a request for a tile and then create vector tiles from OSM just for that tile and then run this as a server.

[davidar]

If you serve vector tiles only you can extract and serve them on a static fileserver. There are examples that do vector tiles serving from S3 (but beware uploading 360 million files to S3 takes forever).

Awesome.

@whyrusleeping Will ingesting that number of files be an issue for us?

Generating vector tiles from the OSM dump just to get a mirror is quite a task. Probably better to not pregenerate all the vector tiles like we do it but serve them dynamically from the specific OSM dump.

For the purposes of this issue, IPFS would be functioning as a distributed CDN. The benefit over traditional CDNs is that it is peer-to-peer, so continues to function even in disconnected/offline networks. This article is a good summary of IPFS if you're interested in more details.

Serving tiles dynamically isn't an option as there is no server. All the processing has to be done either in advance or client-side.

[davidar]

Perhaps we should move this issue to ipfs/apps#25 as it isn't specifically archival-related.