Comments (22)
Since these companies offer tile servers as a paid service, I suspect they aren't going to be willing to share all their secrets :(
Creating vector tiles for the entire world is a big undertaking. At https://github.com/osm2vectortiles/osm2vectortiles we have a completly open process how to generate vector tiles for the entire planet but we dump everthing into a PostGIS database first.
We also looked at tilemaker but ruled it out for scaling to the entire planet. To scale up you need a lot of infrastructure as well (like 6 x 50GB RAM, 16 core servers).
I don't know exactly what the goal of this IPFS project regarding OSM is but we love to collaborate.
If you want to preserve data then vector tiles is perhaps the wrong format - it is not a loss less format.
We love to collaborate. We can also sit together in a Skype call (lukas.martinelli) to discuss questions regarding the entire vector tiles from OSM topic.
from archives.
π this would be soo cool.
from archives.
π
from archives.
Ok, I think we're basically looking for hoverboard by @devtristan, but loading all of its resources from ipfs. @mapbox and @mapzen have also published a lot of useful tools and data.
Relevant links:
- http://blog.mathieu-leplatre.info/serve-your-map-layers-with-a-usual-web-hosting-service.html
- http://gis.stackexchange.com/questions/125037/self-hosting-mapbox-vector-tiles
from archives.
π i would looooove love love this. I think we can do this very easily now. once ipld json-ness lands, this will be a trivial import (yay JSON!)
from archives.
The major obstacle is converting the raw data dump into a format that the js viewers can read. I haven't had much luck with this yet, is anyone more familiar with osm able to lend a hand? :)
from archives.
And the winner is tilemaker by @systemed :)
Here's a semi-functional OSM viewer on IPFS! It still loads some external js, but all the tiles are hosted on ipfs :)
Edit: The downside with tilemaker is that (according to the README)
It's best suited to city and region extracts
so, I still need to work out out to scale it up to the whole planet...
from archives.
@davidar this is really great! i've been waiting so long for this. π
from archives.
interesting, have some go code for tile caches and osm stuff. will dig up.
from archives.
@davidar would be nice if the OSM viewer accepted a query params to specify things like the data root. (i've been using <url>#<ipfs-path>
but #
is a bad idea because it prevents regular anchor action. i now think ?somekey=<ipfs-path>
is better
from archives.
https://github.com/qedus/osmpbf is pretty good , i've used if before.
I think general GIS storage in IPFS will create some interesting options.
from archives.
Hmm, from here about generating vector tiles:
We (in Nutiteq) spent many man-months to optimize it and to get it right for OSM data, and as far as I've heard from Skobbler and Mapbox guys they have done the same.
Since these companies offer tile servers as a paid service, I suspect they aren't going to be willing to share all their secrets :(
The most I've been able to find out about how mapbox generates their tiles is from a single slide in that video I linked earlier (@16min), transcript below:
# headless export
npm install mapnik mbtiles \
tilelive tilelive-bridge
./node_modules/tilelive/bin/tilelive-copy \
bridge:///Users/dane/data/world-borders.tm2source/data.xml \
--bbox=-180,-85,180,85 \
export.mbtiles
# data.xml is created by Mapbox Studio
The problem is that Mapbox Studio uses way to much memory for me to even consider trying to use it to generate worldwide tiles myself.
However, I also found @opensciencemap, who have a very cool vector tile viewer. As a a non-profit research project, hopefully they might be more open to collaboration. I'll try floating the idea with them.
Edit: Also @mapsforge, but it appears to be targeted at android
from archives.
@davidar i have a few machines with a lot of memory, let me know if you want me to do anything for you
from archives.
@davidar happy to help out with understanding & creating vector tiles. The main design decision you need to make first on this task is whether you're creating vector tiles for people to use as basemaps, or as data. For instance, the Mapbox Streets tileset is for visualization, and the OSM QA tiles are for data and analysis. If you're creating tiles for visualization, they can't practically contain all data, and will need to bake in assumptions about what is shown at each zoom level and what map designs are appropriate. These assumptions vary between different maps, hence why there isn't one universal way to 'convert' OSM data to visualization-ready vector tiles.
from archives.
@tmcw that would be brilliant :). My immediate interest is in visualisation (but I suspect @zignig would be very interested in the data analysis side). It would be great to have enough data to do things like routing, but that's not an immediate priority. I don't have any specific requirements other than:
- something that looks better than the crappy demo I made (linked a couple of comments ago)
- reasonable file sizes (preferably the total size should be not significantly larger than the raw osm data)
I'd very much appreciate it if you were able to offer guidance as to how to achieve this. Working with geo data isn't exactly my area (outside of geostats stuff like kriging, etc).
from archives.
@zignig We are investigating using IPFS as an archival option for geodata in @TransforMap. But this is not the OSM topic anymore and rather links to a world of distributed geodata. You are always invited for sidetracking to join our https://discourse.transformap.co
from archives.
@almereyda we'd love to help however we can. maybe open a different issue in this repo?
from archives.
CC: @lukasmartinelli @manuelroth
from archives.
@lukasmartinelli Glad to hear you're interested in collaborating :)
The goal of this particular issue is to get an OSM viewer running on IPFS. Lossiness isn't much of a concern as we can already just mirror the raw OSM dumps.
The major constraint is that all the tiles need to be stored in a static filesystem structure, without any reliance on server-side processing (ie. tileserver-php
isn't an option). Is this likely to be problematic?
from archives.
The major constraint is that all the tiles need to be stored in a static filesystem structure, without any reliance on server-side processing (ie. tileserver-php isn't an option). Is this likely to be problematic?
If you serve vector tiles only you can extract and serve them on a static fileserver. There are examples that do vector tiles serving from S3 (but beware uploading 360 million files to S3 takes forever). You cannot do some features like supporting a filter zoom level or supporting other schemes like TMS but it works.
The goal of this particular issue is to get an OSM viewer running on IPFS. Lossiness isn't much of a concern as we can already just mirror the raw OSM dumps.
Generating vector tiles from the OSM dump just to get a mirror is quite a task. Probably better to not pregenerate all the vector tiles like we do it but serve them dynamically from the specific OSM dump.
Would be cool if tilemaker can handle a request for a tile and then create vector tiles from OSM just for that tile and then run this as a server.
from archives.
If you serve vector tiles only you can extract and serve them on a static fileserver. There are examples that do vector tiles serving from S3 (but beware uploading 360 million files to S3 takes forever).
Awesome.
@whyrusleeping Will ingesting that number of files be an issue for us?
Generating vector tiles from the OSM dump just to get a mirror is quite a task. Probably better to not pregenerate all the vector tiles like we do it but serve them dynamically from the specific OSM dump.
For the purposes of this issue, IPFS would be functioning as a distributed CDN. The benefit over traditional CDNs is that it is peer-to-peer, so continues to function even in disconnected/offline networks. This article is a good summary of IPFS if you're interested in more details.
Serving tiles dynamically isn't an option as there is no server. All the processing has to be done either in advance or client-side.
from archives.
Perhaps we should move this issue to ipfs/apps#25 as it isn't specifically archival-related.
from archives.
Related Issues (20)
- Datasets hosted by cloud providers
- mirrors-instructions have outdated / wrong instructions HOT 1
- KinoΓ―t IPFS-Pack (KinoKabaret Caen) HOT 1
- Software Heritage HOT 1
- Chellenge: 22.5 Trillion Digits of Pi, and Prime Numbers up to one Trillion HOT 2
- Old Internet Files
- Making sure archives are fetchable HOT 11
- Deprecate archives.ipfs.io? HOT 13
- High-impact scientific papers
- Waves in plasmas HOT 1
- Geocities HOT 3
- Congressional Research Service reports (scientific + climate papers) HOT 1
- Godot downloads
- Talks from Our Networks 2018
- It is possible to replace LOCKSS with IPFS/Archives? HOT 1
- LibriVox, free public domain audiobooks HOT 1
- Twitter Datasets on IPFS
- basic works of aristotle
- 10.1016/j.tsf.2007.04.165
- greek literature
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from archives.