mapbox / tilelive-omnivore Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 11.0 263 KB

Implements the tilelive api for a variety of data sources

License: ISC License

JavaScript 100.00%

tilelive-omnivore's People

Contributors

Stargazers

Watchers

Forkers

digidem jingsam rpieciorak testbigorg rubythonode aiyi2099 perliedman rdmurphy isabella232 mapclone

tilelive-omnivore's Issues

What output format?

For most of the data that mapnik-omnivore digests, we'll be generating vector tiles, and hence the map format in mapnik XML should be pbf. What about rasters? Currently this lib hardcodes to webp but is that the right choice?

cc @GretaCB @yhahn

More responsive and detailed tests

This ticket serves as a reminder for the future...

Per chat with @rclark , tests should be a bit more verbose. This will come in handy when gut-checking major changes to the metadata object, and will be relevant in the coming months when we do a major overhaul of the metadata json.

Add more detailed assertions for info returned and for expected mapnikXML generated.
Remove the hardcoded fixtures, and create fixtures on the fly based on metadata returned from mapnik-omnivore.

Do not set a `buffer-size` for raster layers

replace() function on layername

The following error was traced back to this replace function...

## ddd51a1b-1bc4-4666-83b5-d1b65c457cc8

[Tue, 09 Dec 2014 08:02:24 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] Watchbot received message ddd51a1b
[Tue, 09 Dec 2014 08:02:24 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] Stack: unpacker-sidecar-Watchbot-ZDOS8481WBWH
[Tue, 09 Dec 2014 08:02:24 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] InstanceId: i-5ededfb4
[Tue, 09 Dec 2014 08:02:25 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] InstanceDNS: ec2-54-227-94-86.compute-1.amazonaws.com
[Tue, 09 Dec 2014 08:02:25 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] ReceiptHandle: AQEBva6ssDGJ/Ej88MD3y6hsCPBYda2qovuFzzP6/JHyBiZlb4XkY3taYQMo00LOd5OhBFR2OHTKQlgP+kYWlQTiHeou7Iu9EN1b8xcaCKwHwY7VaDl0VEathh85FcPTq+i5RElQ2k+xb1T8+Vza2zyA8mzw7txxsQ7NlPsIcwTrLMYAeo1pH3ZoS4kRbauzKDs/4IOI48CycZMXxuBVuOqS1WPRdgsluNZQhdZB8FYFgy3nrXukAunBPH/CJ71Qhtk0SeUzxaoRnYMpVbhUv9T5BU7MLCfjiOR4kVdsjTh2/rZijwybkqI0HJ3zy8KAieamaMtVIAi/Ppm7iobuVfELFZ5SiKhMQzelHbUfTo6Wi4qy24GK4cmnCnxCxHFXMDT8
[Tue, 09 Dec 2014 08:02:25 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] SentTimestamp: 1418112144716
[Tue, 09 Dec 2014 08:02:25 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] ApproximateFirstReceiveTimestamp: 1418112144717
[Tue, 09 Dec 2014 08:02:25 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] ApproximateReceiveCount: 1
[Tue, 09 Dec 2014 08:02:25 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] Subject: map.upload.created
[Tue, 09 Dec 2014 08:02:25 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] Message: {"id":"f86725546bfa8f79a1f2bd02ce3ddcd6","owner":"cognitio","data":"cognitio.sht49529","url":"http://tilestream-tilesets-production.s3.amazonaws.com/_pending/cognitio/a39af01a87a8fbf15d9b600e9f85d2ce","parts":1,"part":0}
[Tue, 09 Dec 2014 08:02:26 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] [info] Checked if upload job already errored: false
[Tue, 09 Dec 2014 08:02:26 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] [info] Downloaded file from http://tilestream-tilesets-production.s3.amazonaws.com/_pending/cognitio/a39af01a87a8fbf15d9b600e9f85d2ce to /tmp/5bb4178d8d9f0922b656ed94be74f319
[Tue, 09 Dec 2014 08:02:27 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] [info] File type detected: kml
[Tue, 09 Dec 2014 08:02:27 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] [error] File failed validation
[Tue, 09 Dec 2014 08:02:27 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] [error] OGR Plugin: cannot find layer by name 'Nationalpark-Grenze_Kellerwald_kml' in dataset '/tmp/5bb4178d8d9f0922b656ed94be74f319'  encountered during parsing of layer 'Nationalpark-Grenze_Kellerwald_kml' in Layer
[Tue, 09 Dec 2014 08:02:28 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] [info] Reported errored part to api-core
[Tue, 09 Dec 2014 08:02:28 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] [info] Cleaned up original file
[Tue, 09 Dec 2014 08:02:28 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] Service exit code 0
[Tue, 09 Dec 2014 08:02:28 GMT] [ddd51a1b-1bc4-4666-83b5-d1b65c457cc8] Deleted message from the queue

This replace function was used initially due to rasters not displaying and not returning tiles from the getTile function via tilelive-bridge, unless the raster's layername no longer contained the .

I tested a few tifs without the replace function, and both tilelive-omnivore and getTile() were working fine. The initial raster tested that triggered using the replace function might've been a edge case.
@rclark do you remember what tif file that was? We can use it as a test-case.

Layer name overrides

We should allow the caller to override layer names in output vector tiles for certain types of files. This will not pertain to file types with multiple internal layers (KML, GPX, mbtiles), but will apply to other file types.

The order of selection ought to be:

layer is named based on the provided layerName
for geojson, csv, tif, default to the file's name (with extension stripped) if no override is provided
for shapefiles, default to the intrinsic name provided by the data

Zipped shapefile not support by mapnik

I failed to exec

tilelive-copy omnivore://./a.zip mbtiles://./b.mbtiles

After some study, found that mapnik can not generate vector tiles with zipped shapefile, so we need to drop support zipped shapefile to avoid confusion.

Create single tileset for multiple files

Allow tilelive-omnivore to create a single tileset for not only bundled and converted geojson layers, but also multiple files in general. Will require some experimenting here, and some changes from within mapbox-tile-copy as well (mapbox-tile-copy will be passing in the new array of files). But separating into its own bubble here to help modularize the thought process.

Tilelive-omnivore will need to:

allow an array of files as uri param.
call mapnik-omnivore against each file, and somehow translate each metadata object as a separate layer
create a separate Mapnik XML Layer for each file
for now, only allow multiple files with the same filetype (can verify using mapnik-omnivore output)

cc @BergWerkGIS @rclark

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble